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Abstract 

Current  methods  for  monitoring  the  performance  of  Department  of  Defense 
(DOD)  software  development  contractors  have  not  been  successful  in  reversing  the 
current  trend  of  over  budget  and  behind  schedule  software  development.  The  DOD  has 
adopted  the  Software  Engineering  Institute’s  (SETs)  Capability  Maturity  Model  (CMM) 
as  a  method  of  determining  the  process  maturity  of  a  software  developer  with  the  idea 
that  a  more  mature  process  will  lead  to  improved  cost  and  schedule  performance.  The 
goal  of  this  research  was  to  determine  if  a  model  based  on  the  CMM  rating  level  of  a 
contractor  could  be  developed  and  used  in  conjunction  with  statistical  process  control  to 
determine  if  contractor  performance  was  progressing  in  a  satisfactory  manner. 

To  investigate  this  possibility  descriptive  statistics  were  applied  to  historical 
contractor  performance  data  and  a  model  was  established.  A  different  set  of  historical 
data  was  then  used  to  evaluate  the  performance  of  the  new  model.  This  performance  was 
then  compared  to  the  performance  of  current  methods  of  statistical  control. 

The  results  obtained  in  this  research  suggest  that  using  the  CMM  rating  level  of  a 
contractor  to  set  statistical  control  bounds  is  as  good,  and  perhaps  better  than,  the  current 
method  being  employed. 
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A  PRELIMINARY  STUDY  OF  USING  THE  SETS 
CAPABILITY  MATURITY  MODEL  TO  SET  STATISTICAL 
CONTROL  BOUNDS  ON  DOD  CONTRACTOR  COST  AND 
SCHEDULE  PERFORMANCE 

1.  Introduction 


1.1  General  issue 

Weapon  systems  acquired  by  the  Department  of  Defense  (DOD)  in  the  late  1950’s 
and  1960’s  were  comprised  mostly  of  hardware.  Software  played  a  small  role,  if  any,  in 
the  acquisition  of  weapon  systems.  Things  have  changed;  Brown  notes  that  the  DOD  has 
a  “deep  dependence  on  software  for  virtually  all  its  systems”  (Brown,  1996:7). 

“Software  has  become  a  major  cost,  schedule,  and  performance  driver  for  virtually  all 
DOD  weapons,  command  and  control,  and  information  systems”  (Porter,  1994).  This 
deep  reliance  on  software  poses  a  dilemma  for  the  DOD.  Late  and  over  budget  software 
procurements  are  well-known  as  large-scale  software  problems  (Brown,  1996:7). 
Unfortunately,  many  previous  studies  have  identified  numerous  possible  solutions  yet 
most  remain  unimplemented  (Defense  Report,  1987). 

In  an  effort  to  address  the  problem  of  over-budget  and  late  software,  the  DOD 
established  the  Software  Engineering  Institute  (SEI)  in  1984.  SEI  decided  to  attack  the 
problem  by  focusing  on  the  quality  of  the  software  development  process.  This  decision 
was  based  on  the  process  management  principle  which  states  that  “the  quality  of  a 
product  is  largely  governed  by  the  quality  of  the  process  used  to  build  it”  (Paulk,  1997: 
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5).  SEI  designed  a  model  to  measure  an  organization’s  software  development  process 
maturity.  This  model,  the  Capability  Maturity  Model  (CMM),  measures  an 
organization’s  maturity  by  evaluating  process  areas  key  to  software  development.  These 
key  areas  include,  but  are  not  limited  to,  project  planning,  quality  assurance,  product 
engineering,  configuration  management  and  process  management  (Paulk  et.  al.,  1993). 
The  CMM  is  a  framework,  or  road  map,  that  an  organization  can  follow  to  assess  its  own 
software  capability  maturity.  It  can  also  be  used  by  an  outside  agency  to  evaluate  a 
potential  software  developer’s  maturity.  The  organization  maturity  level  is  expressed  by 
an  ordinal  scale  from  1  (lowest)  to  5  (highest)  described  in  Table  1.1.  The  higher  an 
organization’s  maturity  level,  the  more  likely  it  is  to  produce  higher  quality  software. 


Table  1.1  CMM  Level  Description  (Paulk  et.  al.,  1993) 


CMM  Level 

Description 

1  -  Initial 

The  software  process  is  characterized  as  ad  hoc,  and  occasionally  even 
chaotic.  Few  processes  are  defined,  and  success  depends  on  individual 
effort. 

2  -  Repeatable 

Basic  project  management  processes  are  established  to  track  cost, 
schedule,  and  functionality.  The  necessary  process  discipline  is  in 
place  to  repeat  earlier  successes  on  projects  with  similar  applications. 

3  -  Defined 

The  software  process  for  both  management  and  engineering  activities  is 
documented,  standardized,  and  integrated  into  a  standard  software 
process  for  the  organization.  All  projects  use  an  approved,  tailored 
version  of  the  organization’s  standard  software  process  for  developing 
and  maintaining  software. 

4  -  Managed 

Detailed  measures  of  the  software  process  and  product  quality  are 
collected.  Both  the  software  process  and  products  are  quantitatively 
understood  and  controlled. 

5  -  Optimizing 

Continuous  process  improvement  is  enabled  by  quantitative  feedback 
from  the  process  and  from  piloting  innovative  ideas  and  technologies. 
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Lloyd  K.  Mosemann  II,  former  deputy  assistant  secretary  of  the  Air  Force  for 
communications,  computers,  and  logistics  (SAF/AQK),  believes  SEI’s  CMM  to  be  a  step 
toward  solving  the  problems  plaguing  the  development  of  DOD  software  (Mosemaim, 
1992:4).  By  following  the  CMM  road  map,  DOD  procurement  agents  can  assess  a 
potential  software  developer’s  process  maturity,  and  thus  the  likelihood  of  obtaining  a 
quality  software  product  on  time  and  within  budget.  In  1 996,  the  Airlie  Council, 
comprised  of  software  industry  experts,  identified  nine  commercial  best  practices  that 
lead  to  quality  software  development.  One  of  these  practices  is  formal  risk  management 
(Basili  et.  al.,  1997).  Part  of  risk  management  is  attempting  to  reduce  the  risk  involved 
with  a  project.  “Risk  involves  choice,  and  the  imcertainty  that  choice  itself  entails 
(Charette,  1989:  49);  so  it  follows  that  increasing  predictability,  and  thereby  reducing 
uncertainty,  would  be  a  step  towards  reducing  risk  and  increasing  the  quality  of  a 
software  product.  Another  practice  recognized  by  the  Airlie  council  is  the  use  of 
quantitative  targets,  or  statistical  control  bounds,  to  monitor  performance.  This  research 
asserts  that  prediction  intervals,  based  on  the  CMM  rating  level  of  a  contractor,  can  be 
developed  and  used  as  control  bounds  for  cost  and  schedule  performance  of  a  contractor. 
The  key  assumption  is  that  minimum  and  maximum  cost  and  schedule  performance 
ranges  can  be  predicted  from  the  CMM  rating  level  with  some  level  of  confidence,  and 
that  these  intervals  are  reasonable  control  bounds  for  performance  of  a  developer  at  a 
particular  CMM  level. 
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1.2  Specific  Problem 


Recent  research  has  established  a  positive  correlation  between  CMM  rating  and 
the  success  of  software  product  development  in  terms  of  cost  and  schedule  performance 
(Flowe  &  Thordahl,  1994).  It  was  stated  in  that  study  that  a  predictive  model  for  contract 
performance  based  on  CMM  rating  level  may  well  be  of  interest  to  the  software 
development  community  as  a  whole.  However,  little  empirical  research  has  been  done  to 
establish  prediction  and  confidence  intervals  for  cost  and  schedule  performance  based  on 
CMM  rating  level,  not  because  of  a  lack  of  interest,  but  because  of  a  lack  of  available 
data.  Case  studies,  involving  retum-on-investment,  have  been  performed  by  Raytheon, 
Hughes,  and  Oklahoma  City  ALC,  all  level  2  or  3  organizations;  however,  these  studies 
do  not  address  how  this  retum-on-investment  can  be  used  by  DOD  agents  to  predict 
performance.  Bollinger  (1991)  claims  that  “...  it  appears,  unlikely  that  such  [CMM] 
ratings  have  any  meaningful  correlation  to  the  actual  abilities  of  organizations  to  produce 
...  software  on  time  and  within  budget”  (Bollinger  &  McGowan,  1991 :26).  Clearly,  an 
investigation  into  the  predictive  capability  of  the  CMM  model  is  warranted. 

1.3  Research  Objective 

This  follow-on  study  to  Flowe  &  Thordahl’s  1994  research  is  proposed  to  extend 
our  ability  to  predict  intervals  for  software  developer  cost  and  schedule  performance 
based  on  the  developer’s  software  process  maturity  as  determined  by  SETs  CMM  rating 
level  (Flowe  &  Thordahl,  1994:6-6).  This  research  also  proposes  that  an  extended  ability 
to  predict  performance  based  on  CMM  level  can  be  used  to  statistically  control  the 
development  process.  Without  this  extension  of  research,  the  very  basic  notion  that 
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unique  CMM  levels  lead  to  unique  levels  of  performance,  a  fundamental  underpinning  of 
theory,  will  remain  unverified.  For  the  purpose  of  this  study,  performance  will  be 
expressed  in  terms  of  two  measures:  1)  Cost  Performance  Index  (CPI),  a  ratio  of 
Budgeted  Cost  of  Work  Performed  (BCWP)  to  Actual  Cost  of  Work  Performed  (ACWP) 
and  2)  Schedule  Performance  Index  (SPI),  a  ratio  of  BCWP  to  Budgeted  Cost  of  Work 
Scheduled  (BCWS). 

1.4  Scope/Limitations 

The  research  methodology  used  was  chosen  to  yield  the  best  opportunity  of 
achieving  the  objectives  of  this  research,  within  the  time  and  resource  constraints  placed 
on  it.  Also,  the  methodology  chosen  was  consistent  with  that  used  by  Flowe  &  Thordahl 
(1994)  to  maintain  a  consistent  research  approach.  Based  on  these  constraints,  an  already 
existing  database  from  the  previously  mentioned  study  was  used  for  this  effort.  The 
database  consisted  of  organizations  that  met  the  following  criteria: 

a.  Developed  software  for  the  DOD 

b.  Rated  in  accordance  with  the  SETs  CMM  framework 

c.  Tracked  cost  and  schedule  in  a  structured  format 

d.  Reported  cost  and  schedule  data  to  the  DOD 

The  above  constraints  led  to  focusing  on  DOD  contractor  organizations  that 
provided  software  to  Air  Force  Program  Offices  at  the  Aeronautical  Systems  Center 
(ASC)  and  the  Electronics  Systems  Center  (ESC),  where  the  necessary  data  was  reported 
as  part  of  the  Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  contract  requirements. 
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1.5  Overview 


This  research  is  planned  to  establish  a  predictive  model  for  cost  and  schedule 
performance  derived  from  the  SETs  CMM  rating  level  of  the  developers,  and  then  to 
validate  this  predictive  model  as  a  method  to  set  statistical  control  bounds  on  developer 
performance.  This  is  achieved  by  applying  descriptive  statistics  methods  to  information 
obtained  from  the  database  comprised  of  contractor  reported  statistics  to  establish 
prediction  intervals;  and  then  comparing  the  performance  of  a  contractor  to  these  bounds, 
to  see  if  the  intervals  accurately  predict  typical  performance.  The  dependent  variables 
used  in  this  study  are  cost  and  schedule  performance  indices.  Taking  into  account  the 
limitations  and  constraints  under  which  this  research  is  accomplished,  this  study  should 
provide  a  useful  tool  that  the  acquisition  manager  can  use  to  monitor  the  cost  and 
schedule  performance  of  a  contractor.  The  tool  will  provide  early  detection  of 
unsatisfactory  performance,  thus  reducing  the  cost  and  schedule  performance  risk 
associated  with  a  software  product  procurement. 
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2.  Literature  Review 


2.1  Introduction 

Recognizing  the  negative  trends  that  had  emerged  in  the  quality  of  software 
products  being  developed  in  the  DOD,  Lloyd  K.  Mosemann  made  the  CMM  the  focus  of 
a  software  process  improvement  initiative.  He  issued  three  challenges  to  all  Air  Force 
software  development  organizations:  1).  Complete  SEI  CMM  assessments  by  October  1, 
1994,  2).  Perform  follow-up  assessments  every  two  years,  and  3).  Achieve  CMM  level  3 
by  1998  (Coffman  &Thompson,  1997).  This  was  SAF/AQK’s  attempt  to  reverse  the 
trends. 

The  first  two  sections  of  this  literature  review  look  at  the  software  development 
process  and  current  strategies  to  implement  the  process.  The  third  section  takes  an  in- 
depth  look  at  the  SEI  CMM,  including  its  applications  and  limitations.  The  fourth  section 
reviews  some  current  alternatives  to  the  CMM.  The  fifth  section  introduces  common 
performance  measures.  The  sixth  and  seventh  sections  look  at  evidence  suggesting  the 
usefulness  of  the  CMM  rating  level  as  a  predictor  of  performance.  Finally,  the  eighth  and 
last  section  discusses  the  concept  of  statistical  process  control. 

2.2  The  Software  Development  Process 

According  to  Watts  Humphrey,  a  software  development  process  is  “the  set  of 
tools,  methods,  and  practices  we  use  to  produce  a  software  producf’  (Humphrey,  1989). 

In  short,  anything  that  goes  into  converting  inputs  into  a  software  product  is  part  of  the 
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software  development  process.  Having  a  process  is  not  sufficient  to  develop  software; 
however,  one  needs  to  know  how  to  put  the  resources  together.  That  is  where  the 
software  process  model,  or  strategy,  comes  into  play. 

2.3  Program  Strategies/Process  Models 

As  a  follow-up  to  the  software  development  process,  several  development 
paradigms  have  been  popular  at  different  times.  Whereas  the  software  development 
process  provides  the  necessary  building  blocks  to  build  the  software,  the  program  strategy 
provides  a  framework  into  which  these  blocks  fit.  Its  main  purpose  is  to  determine  the 
order  of  the  steps  involved  in  developing  software  (Boehm,  1988).  It  helps  guide  an 
organization,  in  an  orderly  manner,  through  the  development  process.  Program  strategies 
often  address  the  questions  of  “What  to  do  next?”  and  “How  long  shall  we  continue  to  do 
it?”.  Several  models  have  evolved  since  the  earliest  days,  eind  have  been  popular  at 
different  times.  In  the  next  few  segments,  the  more  prominent  ones  will  be  discussed; 
they  include  Code-And-Fix,  Waterfall,  Prototyping,  Evolutionary/Incremental,  and 
Spiral. 

2.3.1  Code  and  Fix. 

This  first  methodology  is  best  described  as  a  haphazard  approach  to  development. 
Developers  using  this  strategy  jump  into  coding  early,  without  fully  thinking  through  the 
problem.  Later,  when  the  requirements  are  better  understood,  they  go  back  and  fix  the 
code  to  reflect  this  understanding.  The  problem  with  this  strategy  is  that  much  time  is 
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wasted  on  rework.  It  may  seem  like  progress  is  being  made,  but  in  reality  the 
programmers  are  only  spinning  their  wheels  (Humphrey,  1989;  7). 


Figure  2-1  The  Waterfall  Model  (Pressman,  1992) 

2.3.2  The  Waterfall  Model. 

Probably  the  most  widely  used  and  well  known  process  model,  the  waterfall 
method,  was  developed  in  the  early  1970’s  by  Royce.  This  model  is  characterized  by  “a 
systematic,  sequential  approach  to  software  development  that  begins  at  the  system  level 
and  progresses  through  analysis,  design,  coding,  testing  and  maintenance”  (Pressman, 
1992:24-26).  Feedback  is  available  at  each  of  the  levels  of  the  waterfall,  tying  back  to 
each  of  the  previous  levels  (refer  to  Figure  2-1).  This  allows  the  developer  to  correct 
problems  in  the  earlier  stages,  that  were  found  later  in  the  development  process.  Several 
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criticisms  during  the  past  ten  years  have  raised  doubt  as  to  the  applicability  of  this  model 
to  all  situations.  Some  of  the  problems  encountered  are  as  follows;  1)  Projects  seldom 
follow  a  smooth  sequential  flow;  most  have  some  type  of  iteration,  2)  This  model 
requires  explicit  requirements  statements,  which  are  rarely  available  at  the  onset  of  a  new 
development,  and  3)  The  customer  does  not  see  a  working  product  until  very  late  in  the 
project,  requiring  great  patience  and  confidence  on  the  part  of  the  customer.  Despite 
these  very  real  problems,  this  model  still  has  an  important  place  in  software  engineering 
(Pressman,  1992:26). 

2.3.3  Prototyping. 

Prototyping  has  become  popular  recently  because  it  addresses  some  of  the 
concerns  dealing  with  the  waterfall  model.  Prototyping  is  the  process  of  developing  a 
working  model  of  the  software  project  to  be  built  (Pressman,  1992:27).  Often  users  are 
not  exactly  sure  what  they  want,  but  they  ’ll  know  it  when  they  see  it.  Prototyping  allows 
the  user  to  get  a  preview  of  the  final  product,  giving  them  a  chance  to  confirm  their 
desires  and  solidify  their  requirements.  Prototypes  are  divided  into  two  categories, 
“throwaway”  and  “evolutionary.” 

Throwaway:  This  category  of  prototypes  is  consistent  with  Fred  Brooks’  maxim, 
“plan  to  throw  one  away;  you  will,  anyhow”  (Brooks,  1996).  The  idea  is  that  the 
prototype  is  only  a  means  to  an  end.  When  the  requirements  are  solidified  and  the 
technical  feasibility  established,  the  prototype  is  discarded  and  the  deliverable  product  is 
started. 
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Evolutionary:  The  idea  behind  evolutionary  is  to  use  all,  or  part  of  the  prototype 
in  the  final  version  of  the  product  (Gordon  &  Bieman,  1994).  By  doing  this,  the  actual 
coding  and  other  work  that  goes  into  developing  the  prototype  is  not  wasted  and  the  time 
and  resources  to  develop  the  deliverable  is  less. 

Some  caution  should  be  used  when  using  the  prototyping  model,  especially  the 
throwaway.  When  a  developer  comes  imder  pressure,  both  schedule  and  budget,  they 
may  be  tempted  to  include  part  or  all  of  the  throwaway  prototype  in  the  final  product. 

The  problem  in  doing  this  is  that  the  prototype  was  designed  to  be  thrown  away,  thus  the 
structure  and  the  integrity  of  the  prototype  is  suspect  (Gordon  &  Bieman,  1994:93). 

2.3.4  Evolutionary/Incremental. 

The  evolutionary  model  is  the  strategy  of  developing  a  product  in  successive 
increments.  The  idea  behind  this  approach  is  that  by  developing  in  increments,  the 
customer  sees  continual  progress,  while  receiving  a  usable  product  earlier.  Each 
increment  of  the  development  goes  through  the  complete  development  cycle,  including 
test.  By  using  this  approach,  system  integration  test  is  effectively  accomplished  as  the 
product  is  being  developed.  When  the  very  last  increment  is  completed,  the  product  is 
finished.  This  approach  is  often  combined  with  other  models.  It  can  incorporate  the  use 
of  prototyping  in  developing  each  increment,  or  can  be  part  of  a  spiral  development. 

2.3.5  The  Spiral  Model. 

The  spiral  model  was  developed  over  several  years  in  an  attempt  to  solve  some  of 
the  shortcomings  of  earlier  models.  It  can  accommodate  most  previous  models  as  special 
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cases,  thus  retaining  their  benefits,  and  provides  guidance  as  to  which  combinations  of 
previous  models  best  fits  a  given  software  development  situation  (Boehm,  1988).  The 
spiral  model  takes  a  cyclical  approach  to  software  development.  The  development 
process  starts  at  the  innermost  area  of  the  spiral  (refer  to  Figure  2-2)  and  proceeds 
outward  along  the  spiral.  Each  time  the  commitment  partition  is  crossed,  a  review  is 
conducted  and  risks  are  assessed.  At  this  point  actions  are  to  be  taken  to  counteract  any 
risks  (Williams,  1995).  According  to  Boehm,  the  primary  advantage  of  the  spiral  model 
is  that  its  flexibility  accommodates  the  good  features  of  previous  models,  while  its  risk 
driven  approach  avoids  their  difficulties.  There  are  difficulties  in  using  the  spiral  model, 
mostly  due  to  its  immaturity.  These  difficulties  include  matching  the  model  to  contract 
software,  reliance  on  risk  assessment  expertise  and  a  need  for  further  elaboration  of  the 
steps  of  the  model  (Boehm,  1988). 

2.4  The  Capability  Maturity  Model 

The  original  version  of  the  CMM  was  called  the  process  maturity  framework. 
Developed  in  1987  by  Watts  Humphrey,  the  maturity  framework,  along  with  the  maturity 
questionnaire,  was  intended  to  help  the  DOD  identify  areas  where  an  organization’s 
software  process  needed  improvement  (Paulk  et.  al.,  1993;  vii). 

This  framework  later  evolved  into  the  CMM,  Version  1 .0  and  eventually,  as  a 
result  of  feedback  from  the  software  community,  was  revised  and  released  as  Version  1.1 
in  1993.  This  version  of  the  CMM  was  intended  as  a  foundation  to  improve  the  software 
process. 
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Figure  2-2  The  Spiral  Model  (Boehm,  1988) 

In  order  to  improve  one’s  process,  one  must  know  the  current  status  of  the  process 
(Humphrey,  1989:3).  The  CMM  was  designed  to  measure  the  maturity  of  an 
organization’s  development  process  with  the  idea  that  increasing  an  organization’s 
process  maturity  in  stages  would  lead  to  a  higher  quality  product  (Paulk  et.  al.,  1993:5). 

As  described  by  Paulk  in  his  paper  on  the  CMM,  an  organization  with  a  mature 
process  can  be  described  as  possessing  an  organization  wide  ability  for  managing 
software  develepment.  On  the  other  hand,  an  organization  with  an  immature  process 
usually  improvises  during  the  course  of  development  and  often  spends  much  time  “fire 
fighting”  (Paulk  et.  al.,  1993:2). 
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The  CMM  consists  of  five  different  levels  ranging  from  1  (the  lowest  maturity)  to 
5(the  highest  maturity).  The  following  is  a  summary  of  the  five  levels  from  Watts 
Humphrey’s  book,  Managing  the  Software  Process: 

Level  1 :  Labeled  initial,  a  software  process  at  this  level  of  maturity  is  sometimes 
considered  ad  hoc  or  even  chaotic.  Usually  none  of  the  procedures  are  formalized,  and  if 
they  are,  they  are  not  well  known  and  often  abandoned  in  time  of  crisis. 

Level  2:  Labeled  repeatable,  a  process  at  this  level  has  achieved  a  measure  of 
statistical  control  not  present  at  the  initial  level.  This  process  is  stable  and  repeatable  and 
has  rigorous  project  management  of  commitments,  costs,  schedules,  and  changes. 

Level  3:  Labeled  defined,  a  process  in  this  level  is  well  established;  it  is  likely  to 
be  used  in  times  of  crisis  instead  of  discarded.  The  organization  now  has  the  foundation 
to  examine  the  process  and  decide  how  to  improve  it.  Advanced  technology  can  now  be 
introduced. 

Level  4:  Labeled  the  managed  level,  an  organization  at  this  level  will  have 
instituted  a  comprehensive  system  for  obtaining  and  analyzing  measurements.  Because 
this  measurement  gathering  and  analyzing  provides  deep  insight  into  the  process,  it  is 
here  that  the  most  significant  quality  improvements  can  be  made. 

Level  5:  Labeled  optimizing,  this  is  the  ultimate  goal  of  an  organization.  The 
organization  at  this  level  has  such  a  good  foundation  in  place  that  they  can  be  proactive  in 
fine-tuning  their  software  development  process,  and  in  turn,  improve  the  quality  of  the 
products. 

Humphrey  states  that  the  reasons  behind  choosing  these  levels  are:  they 
reasonably  represent  historical  evolution  of  improvement  in  real  companies,  they 
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represent  an  achievable  measure  of  improvement  from  one  level  to  the  next,  they  suggest 
interim  improvement  goals  and  progress  measures,  and  they  make  the  priorities  for 
improvement  obvious  once  an  organization’s  current  status  is  known  (Humphrey, 
1989:5). 


Figure  2-3  The  Key  Process  Areas  by  Maturity  Level  (Paulk  et.  al.,  1993). 


2.4.1  Internal  Structure  of  the  CMM. 

Each  CMM  rating  level  is  broken  down  into  several  key  process  areas,  with  the 
exception  of  level  1.  These  process  areas  “identify  clusters  of  related  activities  that 
achieve  a  set  of  goals  important  to  enhancing  process  capability”  (Paulk  et.  al.,  1993:30). 
The  key  process  areas  associated  with  each  maturity  level  are  shown  in  Figure  2-3.  There 
are  other  processes  besides  the  key  processes  that  are  involved  in  developing  and 
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maintaining  software;  however,  they  have  no  bearing  on  achieving  a  given  CMM 
maturity  level. 


Each  key  process  area  is  broken  down  into  five  common  features.  These  common 
features  indicate  whether  the  implementation  or  institutionalization  of  the  key  process 
areas  is  “effective,  repeatable,  and  lasting”  (Paulk  et.  al.,  1993:37).  They  also  contain  the 
key  practices  that,  when  addressed,  accomplish  the  goals  of  the  key  process  areas.  The 
overall  structure  of  the  CMM  can  be  seen  in  Figure  2-4. 

2.4.2  Applications  of  the  CMM. 

There  are  two  main  ways  in  which  the  CMM  can  be  applied  by  an  organization. 
The  first  is  called  a  software  process  assessment  (SPA)  and  the  second  is  called  a 
software  capability  evaluation  (SCE). 
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The  SPA  focuses  on  the  current  status  of  an  organization’s  software  process,  and 
identifies  priorities  for  improvement.  These  assessments  can  be  performed  by  a  team  that 
is  either  internal  or  external  to  the  organization.  Although  these  assessments  can  be 
performed  by  themselves,  they  are  often  done  in  preparation  for  an  SCE  (Bollinger  & 
McGowan,  1991). 

Whereas  the  SPA  focuses  on  the  current  status  in  order  to  establish  priorities  for 
improvement,  the  SCE  focuses  strictly  on  the  current  capability  for  a  given  project. 

SCE’s  are  performed  by  specially  trained  teams  which  are  external  to  the  organization 
being  evaluated.  These  evaluations  are  often  performed  on  bidders  to  a  project  or  on 
existing  contracts  to  monitor  performance  (Paulk  et.  al.,  1993:44). 

Both  the  SPA  and  the  SCE  have  several  commonalties.  Some  of  these  include 
team  selection,  the  maturity  questionnaire,  analysis  of  the  responses,  site  visits,  and  a  list 
of  team  findings  (Paulk  et.  al.,  1993:45,46).  As  described  above,  however,  the  overall 
purpose  of  the  two  applications  discussed  is  quite  different. 

2.4.3  Limitations  of  the  CMM. 

Despite  the  growing  popularity  and  acceptance  of  the  CMM  as  a  measure  of 
process  maturity,  several  eoncems  and  limitations  to  the  model  have  been  expressed  by 
industry  experts. 

Probably  the  biggest  concern  raised  is  the  inability  of  the  CMM  to  adequately 
discriminate  between  levels  of  process  maturity.  An  organization  must  satisfy  all  key 
process  areas  of  a  maturity  level  to  achieve  that  level  (Paulk  et.  al.,  1993).  This 
requirement  may  cause  a  disconnect  in  the  comparative  rating  of  two  organizations.  An 
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organization  that  satisfies  none  of  the  key  process  areas  would  be  considered  a  level  1 
organization.  An  organization  that  satisfied  many  key  process  areas  should  clearly  score 
higher  than  level  1;  however,  this  may  not  be  true  with  the  CMM.  For  example,  if  a 
company  satisfies  most  of  the  areas  for  level  2  and  all  of  the  areas  for  level  3  they  would 
be  rated  a  level  1  because  of  the  areas  they  did  not  satisfy  (Bollinger  &  McGowan, 
1991:31).  In  this  example,  the  company  that  satisfied  most  of  the  level  2  and  3  key 
process  areas  would  have  the  same  rating  as  that  of  the  company  that  satisfied  none,  yet 
the  first  plainly  has  a  more  mature  process  in  the  spirit  of  the  model. 

Another  concern,  or  limitation,  is  the  flexibility  of  a  company  using  the  model. 
Companies  that  follow  the  CMM  framework  may  fall  victim  to  what  is  called  process 
fossilization.  Fossilization  refers  to  a  process  that  cannot  be  easily  changed  in  any 
significant  way  (Bollinger  &  McGowan,  1991 :39).  In  striving  for  and  achieving  level  5, 
and  organization  will  have  committed  many  resources  and  will  have  implemented  many 
tools  and  procedures  for  collecting  data.  When  a  problem  occurs,  this  data  is  used  as  a 
resource  to  determine  where  in  the  existing  structure  the  problem  exists;  and  fails  to 
recognize  a  problem  with  the  overall  structure  of  the  process  itself  (Bollinger  & 
McGowan,  1991 :39).  This  type  of  data  usage  results  in  only  minor  intra-process  change 
and  an  inflexible  overall  process. 

2.5  Alternative  Means  of  Measuring  Capability 

Because  of  the  limitations  of  the  CMM  mentioned  in  the  previous  section,  some 
alternative  approaches  to  measuring  an  organization’s  software  capability  have  been 
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developed.  They  were  designed  to  be  used  to  evaluate  the  software  process  instead  of  the 
CMM  in  situations  for  which  the  CMM  is  not  fully  appropriate  or  suited. 

One  alternative  to  the  CMM  is  the  Software  Development  Capability  Evaluation 
(SDCE).  The  SDCE  method  was  developed  by  ASC  in  1992  and  is  fully  described  in  Air 
Force  Materiel  Command  (AFMC)  Pamphlet  63-103.  The  SDCE  is  meant  to  be  an 
integral  part  of  the  source  selection  process.  In  fact,  the  members  of  the  SDCE  team  are 
also  members  of  the  Source  Selection  Evaluation  Board  (SSEB)  (Babel,  1997).  The 
overall  purpose  of  this  method  is  to  evaluate  a  potential  contractors  capability  to  develop 
the  proposed  project,  as  opposed  to  the  CMM  which  rates  overall  capability.  The  SDCE 
is  used  to  identify  strengths  and  weaknesses  in  specific  source  selection  areas  as  well  as 
the  contractor’s  commitment  to  follow  their  proposed  process  (Babel,  1997). 

A  second  alternative  to  the  CMM  is  the  Software  Acquisition-CMM  (SA-CMM). 
The  CMM  focuses  on  companies  that  develop  software,  but  does  not  address 
organizations  that  acquire  software  fi'om  other  companies.  Recognizing  a  need  for  a 
model  that  focuses  on  the  process  of  acquiring  new  software,  the  SEI  developed  the  SA- 
CMM  and  published  it  in  1996.  The  purpose  of  the  SA-CMM  is  to  “describe  the 
acquirer’s  or  the  buyer’s  role  in  software-intensive  system  acquisition”  (Kind  & 

Ferguson,  1997).  Similar  to  the  CMM,  the  SA-CMM  defines  five  stages,  or  levels,  of 
maturity  for  the  software  acquisition  process.  These  five  levels  are  summarized  in  Table 
2. 1 .  SA-CMM  is  intended  to  be  used  to  improve  the  acquisition  process  similar  to  the 
way  in  which  the  CMM  is  used  to  improve  software  development  processes  (Kind  & 
Ferguson,  1997).  Because  the  SA-CMM  is  based  on  the  CMM  and  is  very  similar  in 
structure,  it  maintains  the  same  limitations  as  the  CMM. 
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Although  this  is  not  a  conclusive  list,  it  points  out  that  the  CMM  is,  by  no  means, 
the  only  available  method  of  improving  or  evaluating  the  capability  of  a  potential 
contractor.  To  this  point,  however,  the  CMM  appears  to  be  the  most  popular  and  widely 
known  model. 

Table  2.1  The  SA-CMM  Maturity  Level  Description  (Kind  &  Ferguson,  1997) 


CMM  Level 

Description 

1  -  Initial 

The  organization  does  not  have  documented  processes. 

2  -  Repeatable 

Basic  acquisition  management  instills  discipline  at  the  project  level. 

3  -  Defined 

Acquisition  organization-wide  processes  are  defined,  then  tailored  for 
each  project. 

4  -  Quantitative 

Decisions  on  processes  and  products  are  based  on  formal  quantitative 
measures. 

5  -  Optimizing 

Continual  process  and  acquisition  methodology  improvements  occur 
based  on  quantitative  feedback  and  form  piloting  innovative  ideas  and 
technologies. 

2.6  Cost  and  Schedule  Performance  Measures 

The  Airlie  Council,  in  their  study  of  industry  best  practices  in  1996,  recognized 
the  project  control  panel  as  both  a  useful  tool  and  a  concept  for  tracking  the  progress  of  a 
project,  and  predicting  its  future  progress  (Basili  et.  al.,  1997).  The  control  panel  consists 
of  several  measures  of  performance  in  primary  areas  of  a  project;  such  as  productivity, 
completion,  change,  staff,  risk,  and  quality. 

One  measure  of  particular  interest  to  this  research  effort  is  the  Cost  Performance 
Index  (CPI).  This  measure  shows  how  well  a  project  team  is  meeting  its  budget  goals. 
The  CPI  is  a  ratio  of  Budgeted  Cost  of  Work  Performed  (BCWP)  to  Actual  Cost  of  Work 
performed  (ACWP),  two  parameters  present  in  most  Earned  Value  Management  Systems 
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(EVMS’s).  The  CPI  provides  a  historical  measure  of  average  productivity.  A  CPI  of  1 .0 
indicates  a  project  that  is  exactly  on  target  for  budget.  A  value  less  than  1.0  indicates  a 
budget  overrun  vv'here  a  value  greater  than  1.0  indicates  a  budget  underrun. 

Another  performance  measure  of  interest  to  this  study  is  the  Schedule 
Performance  Index  (SPI).  Although  not  present  on  the  project  control  panel  mentioned 
above,  the  SPI  is  a  relative  to  the  CPI.  Where  the  CPI  is  a  historical  measure  of  cost 
performance,  the  SPI  is  a  measure  of  schedule  performance.  The  SPI  is  a  ratio  of  BCWP 
to  Budgeted  Cost  of  Work  Scheduled  (BCWS),  two  parameters  also  present  in  most 
EVMS’s.  Like  the  CPI,  a  value  of  1.0  indicates  an  on  schedule  project.  A  value  less  than 
1 .0  indicates  a  schedule  overrun  and  a  value  greater  than  1 .0  indicates  a  schedule 
underrun. 

CPI  =  BCWP/ACWP  (2.1) 

SPI  =  BCWP/BCWS  (2.2) 

The  above  two  measures  are  not  the  only  measures  of  cost  and  schedule 
performance.  However,  these  two  measures  have  become  standard  for  both  industry  and 
government  (Nicholas,  1990:376-389). 

2.7  Return-On-Investment  Studies 

Several  companies  and  organizations  have  done  retum-on-investment  (ROI) 
studies  showing  the  economic  benefits  of  moving  up  the  CMM  maturity  scale.  The 
studies  identified  the  costs  associated  with  trying  to  improve  one’s  CMM  rating  level. 
They  then  identified  and  assigned  dollar  values  to  the  perceived  benefits,  both  economic 
and  non-economic,  to  determine  the  overall  ROI.  Three  studies  of  prominent 
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organizations  at  different  levels  are  described  in  the  further  detail  in  the  following 
sections. 

2.7.1  Hughes  Aircraft. 

In  1987,  Hughes  Aircraft  employed  a  team  from  the  Software  Engineering 
Institute  (SEI),  at  a  cost  of  $45,000,  to  perform  an  assessment  of  the  Software 
Engineering  Division  (SED)  of  the  company.  The  SED  was  rated  at  a  level  2 
(Humphrey,  Snyder,  and  Willis;  1991 : 13).  After  receiving  the  recommendations  from  the 
assessment  team,  an  action  plan  was  devised  and  implemented  to  improve  the  software 
process.  Over  the  coiurse  of  18  months,  Hughes  expended  78  man-months  of  effort  and  a 
total  cost  of  $400,000  to  implement  the  action  plan. 

When  the  SEI  performed  another  assessment  in  1 990,  it  found  that  the  SED  had 
improved  to  a  strong  level  3.  In  the  course  of  improving  from  level  2  to  3,  several 
benefits  were  realized.  Hughes  found  that  working  conditions,  employee  morale,  and 
project  schedule  and  cost  performance  had  improved.  The  economic  value  of  the 
improvements  was  estimated  to  be  about  $2  million  amiually  (Humphrey,  Snyder  and 
Willis;  1991). 

2.7.2  Oklahoma  City  ALC 

In  1 990,  the  Oklahoma  City  Air  Logistic  Center  (OC-ALC),  Software  Division 
(LAS)  was  rated  by  the  SEI  at  a  CMM  level  of  1 .  In  1 993,  they  were  again  rated  and  had 
achieved  a  level  2.  Also,  in  1993,  and  independent  study  was  conducted  to  determine  the 
cost  of  process  improvement  and  the  benefits  obtained.  The  study  found  that  over  an  8- 


2-16 


year  period,  an  investment  of  $1.5  million  by  LAS  resulted  in  a  cost  savings  of  $1 1.3 
million.  Other  findings  included  a  90%  reduction  in  defect  rate,  a  26%  reduction  in  test 
program  set  (TPS)  maintenance  costs,  and  a  ten  fold  increase  in  productivity 
(Department,  1996:7-35). 

2.7.3  Raytheon. 

In  1988,  an  internal  assessment  of  the  Software  Systems  Lab  at  Raytheon,  based 
on  the  CMM  questionnaire,  rated  the  lab  at  slightly  less  than  level  2.  Four  areas  were 
identified  as  needing  improvement:  documented  practices  and  procedures,  training,  tools 
and  methods,  and  metrics  (Department,  1996:7-40). 

In  1992,  a  follow-up  analysis  revealed  that  Raytheon  achieved  a  7.7:1  ROI(  a 
$4.48  million  return  on  a  $.58  million  investment).  Other  noted  savings  included  a  75% 
reduction  in  rework  since  1988  and  a  230%  increase  in  productivity  (Department,  1996:7- 
41). 

2.8  Correlation  Study  of  the  CMM  and  Software  Development 
Performance 

In  1 994,  Robert  Flowe  and  James  Thordahl  conducted  a  study  examining  the 
correlation  between  CMM  rating  level,  and  cost  and  schedule  performance  of  an 
organization.  Although  based  on  a  relatively  small  database,  the  results  provide  some 
interesting  insights. 

The  research  used  CPI  and  SPI  as  measures  for  performance.  The  study  also 
considered  nine  possible  moderating  variables  when  establishing  correlation.  The  results 
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suggest  that  a  positive  correlation  exists  between  CMM  rating  level  and  both  the  CPI  and 
SPI.  The  research  found  that  a  strong  correlation  is  present  when  the  moderating  variable 
of  “project  relevance”  is  high.  Also,  the  results  reveal  that  the  correlation  with  SPI 
becomes  more  evident  when  the  moderation  variable  of  “percent  complete”  is  taken  into 
consideration  (Flowe  &  Thordahl,  1994:6-2,3). 

2.9  Summary 

The  ROI  studies  described  earlier  provide  insights  into  the  economic  value  of 
moving  up  the  CMM  scale;  however,  they  provide  no  useful  information  about  how  the 
CMM  can  be  used  by  the  software  acquisition  manager.  The  Flowe  and  Thordahl  study 
provides  evidence  supporting  the  idea  that  higher  CMM  levels  indicate  better  cost  and 
schedule  performance;  however,  the  study  stops  short  of  explaining  how  this  correlation 
can  be  beneficial  to  the  software  acquisition  manager. 

This  research  attempts  to  build  upon  the  relationship  between  CMM  and 
performance,  described  in  the  previously  mentioned  studies.  It  proposes  a  method  of 
combining  the  CMM  rating  level  with  the  concept  of  statistical  process  control,  which 
was  developed  in  the  1930’s  and  later  promoted  by  Edward  Deming  and  Joseph  Juran,  to 
produce  a  method  for  the  software  acquisition  manager  to  monitor  and  control  the 
performance  of  a  software  development  contractor  (Paulk  et.  al.,  1993). 
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3.  Methodology 


3.1  Overview 

Once  the  subject  of  this  research,  the  CMM,  was  chosen;  the  research  continued 
in  four  phases.  The  first  phase  was  the  problem  defmition/scope  phase.  During  this 
phase,  a  specific  problem  dealing  with  the  subject  was  selected.  Also,  the  scope  of  the 
problem  was  defined.  The  second  phase  was  the  data  identification/gathering  phase. 
During  this  phase,  the  appropriate  data  was  identified,  located,  and  gathered.  Phase  three 
was  the  model  development  phase.  During  this  phase,  the  data  was  analyzed  and  a 
proposed  model  was  developed.  Finally,  phase  four  was  the  model  validation  phase. 
During  this  phase,  the  proposed  model  was  validated  using  historical  data  gathered  about 
members  of  the  target  population.  The  following  sections  describe  each  of  the  four 
phases  in  full  detail. 

3.2  Problem  Definition/Scope 

The  purpose  of  this  phase  was  to  define  a  specific  research  problem  associated 
with  the  CMM.  A  review  of  the  existing  research  pertaining  to  the  CMM  revealed  that 
research  exploring  the  predictive  nature  of  the  CMM  might  be  useful  to  the  software 
acquisition  community  (Flowe  &  Thordahl,  1994).  It  was  then  necessary  to  define  the 
scope  of  the  research  because  of  the  broad  nature  of  the  problem,  and  the  limited  time  and 
resources  available  to  conduct  the  research.  After  further  review  of  the  existing  literature. 
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the  decision  was  made  to  focus  this  research  on  applying  a  predictive  model,  based  on  the 
CMM,  to  the  statistical  process  control  of  DOD  contractors. 

3.3  Data  Identification  and  Gathering 

Once  the  problem  had  been  defined  and  the  scope  clearly  delineated,  the  research 
moved  into  the  data  identification  and  gathering  phase.  The  first  step  of  this  phase  was  to 
identify  the  data  required  to  conduct  this  research.  CMM  rating  level  was  chosen  to  be 
the  independent  variable.  ACWP,  BCWP,  and  BCWS  were  selected  based  on  the  the 
dependent  variables  of  interest,  CPI  and  SPI. 

The  next  step  was  to  locate  reliable  sources  for  the  required  data.  After  a  search 
of  the  literature,  a  database  containing  secondary  historic  data  from  DOD  software 
development  contracts  that  had  been  established  by  Robert  Flowe  and  James  Thordahl  for 
their  research  was  located  (Flowe  &  Thordahl,  1 994).  Robert  Flowe  was  contacted  and  a 
copy  of  the  database  was  obtained.  The  database  consisted  of  pre-established  contractor 
process  maturity  ratings  (as  defined  by  the  SETs  CMM),  and  cost  and  schedule  data 
reported  to  ASC  and  ESC  in  Cost  Performance  Reports  (CPR’s)  as  part  of  their  contract 
fulfillment.  The  following  is  a  summary  of  the  steps  used  by  Flowe  and  Thordahl  to 
obtain  their  information  (Flowe  &  Thordahl,  1994): 

1)  Identify  appropriate  contract  elements:  During  this  step,  contracts  that 
reported  software  development  costs  as  a  discrete  contract  work  breakdown 
structure  (CWBS)  element  were  identified  in  the  ASC  and  ESC  libraries. 

2)  Determine  rating  of  contractor:  After  identifying  the  appropriate  contracts,  it 
was  necessary  to  establish  whether  the  contractor,  associated  with  each 
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contract,  had  been  rated  using  the  CM  methodology.  If  not,  that  contract  was 
discarded  as  a  possible  source  of  data;  if  they  had,  the  rating  information, 
including  method  used  and  date  rating  was  given,  was  recorded. 

3)  Collection  of  relevant  cost/schedule  information:  During  this  step,  cost  and 
schedule  performance  information,  covering  a  period  of  six  months  prior  to 
and  six  months  following  the  rating  date,  was  collected. 

4)  Collection  of  moderating  data:  Finally,  other  moderating  data  which  may  be 
used  to  characterize  the  software  development  project  was  collected  to  be  used 
to  gain  further  insight  into  the  performance  data  obtained. 

These  steps  are  depicted  in  Figure  3-1 . 


Figure  3-1  Data  Gathering  Flow  Chart  (Flowe  &  Thordahl,  1994) 
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The  reliability  of  the  information  in  the  Flowe  and  Thordahl  database  was 
considered  sufficient  for  the  purposes  of  this  research  because  the  collection,  content,  and 
reporting  of  the  information  are  governed  by  the  C/SCSC  guidelines.  Also,  the  same 
criteria  for  cost  and  schedule  measurement  and  reporting  are  mandated  for  all  contracts, 
making  the  data  obtained  reliable  for  comparison  between  different  contracts. 

An  attempt  was  made  to  add  to  the  validity  of  the  database  by  adding  contractor 
information  fi-om  Space  and  Missile  Systems  Center  (SMC)  contracts.  The  person  in 
charge  of  the  SMC  cost  library  was  contacted,  and  the  contents  of  the  library  was 
discussed.  It  was  learned  that  necessary  data  (contractor  identification)  was  not  kept  in 
the  library.  Because  of  this  fact,  contractor  CMM  rating  level  could  not  be  ascertained 
and  linked  to  the  performance  information,  making  use  of  the  SMC  cost  library 
information  for  this  research  impractical.  Because  there  are  few,  if  any,  reliable  sources 
of  data  it  was  decided  that  the  existing  database  would  be  sufficient,  based  on  the  target 
population  of  this  research  (DOD  contractors). 

Some  of  the  data  points  in  of  the  Flowe  and  Thordahl  database  had  to  be  excluded 
for  this  research  effort.  Low  levels  of  contract  effort  cause  the  variances  of  the  indices 
that  are  more  due  to  lack  of  activity  than  to  actual  variances  in  contractor  performance. 
Flowe  and  Thordahl  calculated  a  ratio  of  contract  activity  during  the  twelve  month  period 
relative  to  total  activity  to  date.  If  this  ratio  showed  a  level  of  activity  of  less  than  1%  for 
any  of  the  three  parameters,  BCWS,  BCWP,  and  ACWP,  the  data  point  was  excluded 
(Flowe  &  Thordahl,  1994).  These  points  are  identified  by  comments  in  the  investigator 
comment  box  of  the  data  forms  in  Appendix  A. 
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One  of  the  moderating  variables  collected  by  Flowe  and  Thordahl  was  rating 
relevance.  This  moderating  variable  rates  the  relevance  of  the  project  listed  in  the  WBS 
to  the  actual  CMM  rating  of  the  organization.  If  this  variable  is  listed  as  high  or  very 
high,  the  project  in  the  WBS  was  the  project  used  to  obtain  the  organization  rating.  In  an 
attempt  to  develop  a  model  that  is  as  accurate  as  possible  in  relationship  to  the  CMM 
rating  level  of  the  contractor,  only  contracts  with  a  rating  relevance  of  high  or  very  high 
were  used  to  develop  the  model. 


3.4  Model  Development 


The  first  step  of  the  data  analysis  phase,  following  the  removal  of  data  to  be  used 
in  the  validation  phase  (validation  data  selection  is  described  in  detail  in  the  next  section), 
was  to  separate  the  data  based  on  CMM  rating  level.  After  separation,  equations  3.1  and 
3.2  were  applied  to  the  data  to  obtain  the  sample  mean  and  standard  deviation  for  each 
rating  level  (Devore,  1995). 
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where:  is  the  sample  mean, 

n  is  the  sample  size, 
x,  is  a  point  in  the  sample. 
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where;  s  is  the  sample  standaird  deviation, 
n  is  the  sample  size. 

Xj  is  a  point  in  the  sample. 

Xbar  is  the  sample  mean. 

The  next  step  was  to  calculate  prediction  intervals,  to  be  used  as  the  predictive 
model  upper  and  lower  statistical  control  bounds  for  performance,  using  equations  3.3 
and  3.4.  An  assumption  of  normality  must  be  made  about  the  data  distributions  for  these 
equations  to  apply  to  this  research  (Devore,  1995).  The  intervals  calculated  using  these 
equations  will  be  known  as  the  model  from  here  on  out. 
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where:  UB  and  LB  are  the  prediction  interval  upper  and  lower  bounds. 
Xbar  is  the  sample  mean. 

t  is  the  value  of  the  t  statistic.(a  =  1  -  prediction  level/100) 
s  is  the  sample  standard  deviation, 
n  is  the  sample  size. 


One  graphical  method  of  validating  an  assumption  of  normality  is  the  box  and 
whisker  plot  (refer  to  Figure  3-2).  A  box  and  whisker  plot  gives  a  quick  graphical  picture 
of  the  median  of  a  sample  distribution  and  the  extent  and  nature  of  any  departure  from 
symmetry  (Devore,  1995).  It  can  also  be  used  to  identify  any  points  that  lie  unusually  far 
from  the  main  body  of  data.  This  method  can  be  used  to  identify  sample  distributions 
that  deviate  severally  from  normal;  however,  for  small  sample  sizes  the  box  and  whisker 
plot  may  be  misleading  and  a  more  precise  method  is  required. 
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A  more  precise  method  of  validating  the  normality  assumption  is  the  Wilk- 
Shapiro/Rankit  Plot  Procedure  (refer  to  Figure  3-3).  It  can  be  used  to  examine  vv^hether 
data  conform  to  a  normal  distribution  or  not  (Analytical,  1996).  This  method  yields  a 
statistic  equal  to  the  square  of  the  linear  correlation  betvv^een  the  rankits  and  the  order 
statistics  (Analytical,  1996).  The  closer  to  1.00  the  value  is,  the  more  normal  the 
distribution  is.  For  a  small  sample,  typically  less  than  twenty  data  points,  a  value 


Figure  3-2  Sample  Box  and  Whisker  Plot  for  a  normal  distribution 

of  .8  or  higher  is  sufficient  for  the  distribution  to  be  approximated  with  the  normal 
(Reynolds,  1997). 

3.5  Model  Validation 

The  research  entered  the  model  validation  phase  following  completion  of  data 
analysis  and  the  development  of  the  model.  The  first  step  of  this  phase  was  to  select  the 
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data  to  be  used  for  validating  the  model.  Since  the  target  population  of  this  research  is 
DOD  software  development  contractors,  it  was  decided  to  select  data  from  contractors 
within  this  population.  An  available  source  of  information  was  the  existing  database. 
Appropriate  contractors  were  selected  from  the  database  based  on  CMM  rating  level  and 
the  number  of  data  points  provided  by  each  contractor.  In  an  attempt  to  obtain  enough 
points  to  do  the  validation  without  reducing  the  database  size  significantly,  contractors 
that  had  provided  three  data  points  were  chosen.  These  contractors  are  identified  by  a 
comment  in  the  investigator  comment  box  of  the  data  forms  in  Appendix  A. 


Wilk-Shapiro  /  Rankit  Plot  of  NORMAL 


Raokits 

Approximate  Wilk-Shapiro  0.9538 

Figure  3-3  Sample  Wilks-Shapiro  Plot  of  a  Normal  Distribution 

CPI  and  SPI  were  calculated  for  each  of  the  points  using  equations  2.1  and  2.2 
respectively.  These  values  were  then  compared  to  the  model  value  for  the  upper  and 
lower  control  bounds  to  determine  which  points  fall  within  the  bounds  and  which  points 
fall  outside. 
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One  current  method  of  determining  whether  a  contractor’s  performance  is 
acceptable  or  not  is  to  calculate  the  cost  and  schedule  variance  percentage  of  the 
contractor’s  performance  and  compare  them  against  set  limits.  A  common  limit  currently 
used  by  managers  is  ±10%  variance  for  both  cost  and  schedule  (Ferens,  1997). 

In  order  for  our  proposed  model  to  be  at  least  as  good  as  the  current  method,  it 
was  expected  that  any  point  with  a  variance  percentage  of  greater  than  ±10%  would  fall 
outside  the  proposed  model’s  control  bounds,  and  any  point  with  a  variance  percentage 
within  the  ±10%  range  would  fall  inside  the  proposed  model’s  control  bounds.  The  cost 
and  schedule  variance  percentages  were  calculated  for  each  point  using  equation  3.5  and 
3.6  respectively. 

Cost  Variance  %(CV%)  =  1 00*(BC WP  -  AC  WP)/BCWP  (3.5) 

Schedule  Variance  %(SV%)  =  1 00*(BCWP  -  BC WS)/BC WS  (3.6) 

These  variance  percentages  were  then  compared  to  the  ±10%  limit  and  used  to  determine 
the  expected  position  of  the  point  with  regards  to  the  model’s  control  bounds.  Finally, 
any  deviation  from  the  expected  position  was  noted. 
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4.  Data  Analysis/Results 


4.1  Model  Development 

Having  separated  the  data  into  two  parts,  the  complete  data  set  consisting  of  all 
data  to  be  used  in  the  development  of  the  model,  and  the  validation  data  set  consisting  of 
the  data  to  be  used  in  validating  the  model,  the  next  step  in  developing  the  model  is  to 
validate  the  assumption  of  normality  for  the  complete  data  set.  The  box  plots  of  the  CPI 
and  SPI  are  inconclusive  (see  Figures  B-1  and  B-2  in  Appendix  B).  There  are  no  highly 
extreme  values  to  suggest  that  the  distributions  are  not  normal,  however,  the  plots  are  not 
exactly  symmetrical  so  further  analysis  is  needed. 

Wilk-Shapiro  Rankit  Plots  were  constructed  for  each  level  of  data  (see  Figures  B- 
3  through  B-8  in  Appendix  B).  The  Wilk-Shapiro  statistics  obtained  from  these  plots  are 
summarized  in  Table  4.1 .  The  values  obtained  are  not  inconsistent  with  normal 
distributions  and  support  the  assumption  of  normality. 


Table  4.1  Wilk-Shapiro  Statistics  for  SPI  and  CPI 


Rating  Level 

SPI 

CPI 

1 

0.87 

0.84 

2 

0.93 

0.91 

3 

0.93 

0.90 

Having  validated  the  assumption  of  normality,  the  next  step  is  to  apply  descriptive 
statistics  to  the  complete  data  set  to  obtain  the  mean  and  the  standard  deviation.  These 
values  can  then  be  used  to  construct  the  prediction  intervals  necessary  to  develop  the 
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model.  Once  again  using  Statistix  for  windows,  the  values  were  obtained  and  are 
summarized  in  Tables  4.2  and  4.3.  Some  of  the  values  are  contrary  to  the  CMM  theory 
which  states  that  as  rating  level  goes  up,  the  performance  of  the  contractor  moves  closer 
to  the  ideal  and  the  variance  improves.  These  discrepancies  are  addressed  in  the 
limitations  section  of  chapter  five. 


Table  4.2  Descriptive  Statistics  for  CPI 


CMM  Rating 

Number  in  Sample 

Mean 

Standard  Deviation 

1 

11 

0.7326 

0.2883 

2 

12 

1.2489 

0.4169 

3 

11 

0.988 

0.1104 

Table  4.3  Descriptive  Statistics  for  SPI 


CMM  Rating 

Number  in  Sample 

Mean 

Standard  Deviation 

1 

11 

1.0668 

0.3454 

2 

12 

0.9741 

0.0531 

3 

11 

1.0457 

0.0891 

These  values  can  now  be  used  to  construct  the  prediction  intervals  for  all  three 
rating  levels  and  both  performance  indices.  The  prediction  level  used  in  this  study  is 
90%.  Usually  a  higher  prediction  level  is  preferred,  but  for  the  size  of  our  sample  a 
higher  prediction  level  would  yield  intervals  too  wide  to  be  meaningful.  The  a 
corresponding  to  a  90%  prediction  level  is  1  -  prediction  level/100  or  .10.  Dividing  a  by 
two  yields  the  required  value  for  equations  3.3  and  3.4  which  is  .05.  The  t-statistic  for 
this  value,  t  os  „.i  can  be  obtained  from  a  standard  table  such  as  the  one  in  Devore  (Devore, 
1995:  707).  Substituting  into  equations  3.3  and  3.4  yields  the  intervals  displayed  in  Table 
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4.4  and  4.5.  Once  again,  one  of  the  results  is  not  consistent  with  CMM  theory  and  is 
addressed  in  chapter  five. 


Table  4.4  CPI  Portion  of  Proposed  Model 


CMM  Rating 

n 

^.05,n-l 

Lower  Bound 

Upper  bound’ 

1 

11 

1.812 

0.186971 

1.278229 

2 

12 

1.796 

0.469574 

2.028226 

3 

11 

1.812 

0.77906 

1.19694 

Table  4.5  SPI  Portion  of  Proposed  Model 


CMM  Rating 

n 

^.05,n-l 

Lower  Bound 

Upper  bound 

1 

11 

1.812 

0.413106 

1.720494 

2 

12 

1.796 

0.874838 

1.073362 

3 

11 

1.812 

0.877072 

1.214328 

The  intervals  in  Table  4.4  and  4.5  constitute  the  proposed  model.  This  model  is 
proposed  to  be  used  by  acquisition  managers  to  predict  performance  or  monitor 
performance  of  a  contractor,  based  on  the  contractors  CMM  level.  A  performance  value 
inside  the  interval  for  a  given  rating  level  denotes  acceptable,  or  typical,  performance  for 
that  level.  A  value  outside  the  interval  depicts  unacceptable,  or  atypical,  performance. 

4.2  Model  Validation 

The  first  step  in  the  model  validation  process  is  to  compare  the  performance 
values  for  the  selected  data  validation  points  to  the  model  interval  bounds  developed 
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during  the  model  development  phase  and  note  whether  the  value  is  inside  the  interval  or 
not.  A  summary  of  the  results  of  this  comparison  is  located  in  Table  4.6. 

The  second  step  in  the  model  validation  process  is  to  calculate  the  CV%  and  the 
SV%  using  equation  3.5  and  3.6  respectively,  and  compare  these  percentages  to  the 
standard  limits  of  ±10%.  The  location  of  the  percentages  (inside  or  outside  the  limits)  is 
then  noted.  A  summary  of  the  results  of  this  comparison  are  located  in  Table  4.7. 


Table  4.6  Comparison  of  CPI  and  SPI  to  Model  Bounds 


Contractor 
Code(Append  A) 

WBS 

Element 

CMM 

Rating 

SPI 

Value 

CPI 

Value 

Inside 
model  SPI 
range? 

Inside 
model  CPI 
range? 

IC 

1 

1 

1.00 

0.87 

yes 

yes 

2 

1 

1.04 

0.56 

yes 

yes 

3 

1 

1.01 

0.84 

yes 

yes 

FA 

1 

2 

0.39 

yes 

no 

GB 

1 

2 

0.35 

no 

no 

HA 

1 

2 

1.05 

0.84 

yes 

yes 

JB 

1 

3 

0.90 

yes 

yes 

2 

3 

1.29 

1.98 

no 

no 

3 

3 

1.31 

2.16 

no 

no 

Table  4.7  Comparison  of  CPI  and  SPI  variance  %  to  ±10%  limits 


Contractor 
Code(Append  A) 

SPI  var 
% 

CPI  var 

% 

Inside  10% 
SPI  limit? 

Inside  10% 
CPI  limit? 

IC 

1 

1 

0.00 

-12.90 

yes 

no 

2 

1 

4.09 

-58.86 

yes 

no 

1 

yes 

no 

FA 

1 

2 

yes 

no 

GB 

1 

2 

7.72 

-175.51 

yes 

no 

HA 

1 

2 

5.66 

-12.88 

yes 

no 

JB 

1 

J 

-10.04 

8.69 

no 

yes 

2 

o 

J 

no 

no 

3 

3 

no 

no 
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The  final  step  in  the  validation  process  is  to  compare  the  results  in  Table  4.7  to  the 
results  in  Table  4.6.  Table  4.8  summarizes  the  comparison  between  the  10%  limit  method 
and  the  proposed  model  method.  A  “yes"’  value  indicates  values  lying  inside  the  limits  for 
the  respective  methods  and  a  “no”  value  indicates  values  lying  outside  the  limits  for  the 
respective  methods.  A  disagreement  is  defined  as  a  difference  between  the  value  in  the 
10%  column  and  the  value  in  the  Model  column.  For  the  SPI  performance  measure,  one 
CMM  rating  level  2  point  disagreed  and  one  CMM  rating  level  3  point  disagreed.  For  the 
CPI  measure,  all  three  CMM  level  I  points  were  in  disagreement  and  one  CMM  level  2 
point.  The  shaded  areas  in  the  table  represent  these  disagreements  between  the  10% 
method  and  the  model  method. 

Table  4.8  Comparison  of  10%  Method  to  Model  Method 


Contractor 

WBS 

Rating 

SPI  for  10% 

SPi  for  Model 

CPI  for  10% 

CPI  for  Model 

IC 

1 

1 

yes 

yes 

jes  hhA,-.' 

2 

1 

yes 

yes 

no 

■■ 

3 

1 

yes 

yes 

no 

■  .'  ■■■yA  yes 

FA 

2 

yes 

yes 

no 

no 

GB  i 

1 

2 

.  yes  . 

. '''m'y 

no 

no 

HA 

1 

2 

iyes 

yes 

-.1 

JB 

1 

•-s 

:> 

. lid 

%:■:  yeS" 

yes 

yes 

2 

no 

no 

no 

no 

.3 

no 

no 

no 

no 

4.3  Analysis  of  Differences 

There  are  several  possible  explanations  for  the  differences  noticed  between  the 
predicted  values  obtained  using  the  current  practice  of  using  ±10%  variance  as  bounds 
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and  the  values  obtained  using  the  proposed  model.  The  following  paragraphs  will  give 
some  of  the  more  probable  explanations. 

For  the  SPI  performance  measure,  two  of  the  nine  points  disagreed.  Both  of  these 
points  disagreed  at  the  second  decimal  point  level.  The  number  of  decimal  places 
reported  in  the  model  intervals  and  also  the  variance  calculations  are  more  a  result  of  the 
programs  used  to  calculate  them,  (Microsoft  Excel®  and  Statistix®),  than  an  indication  of 
significance.  For  this  reason  it  is  possible  that  in  reality  there  is  agreement  between  the 
model  and  the  current  method  being  used. 

For  the  CPI  performance  measure,  all  three  of  the  CMM  level  1  points  disagreed 
and  one  CMM  level  2  point.  The  current  method  assumes  that  all  contractors  should  be 
capable  of  performing  within  the  10%  limits,  it  does  not  take  into  account  the  differing 
maturity  levels  of  the  organization.  According  to  the  CMM,  level  1  organizations  are  ad 
hoc  and  have  a  high  variance  (Paulk  et.  al.,  1993).  Because  of  this,  the  model  intervals 
for  CMM  level  1  contractors  are  extremely  wide,  causing  points  that  are  outside  the  1 0% 
limits  to  still  be  within  the  acceptable  performance  levels  for  a  typical  CMM  level  1 
organization.  Another  possible  explanation  is  due  to  the  sample  size  for  the  model 
development.  The  sample  is  relatively  small  in  this  preliminary  study  causing  the  t- 
statistic  to  be  rather  large.  This  will  cause  the  intervals  to  be  wide  and  might  explain  why 
the  model  says  that  the  contractors  performance  is  acceptable,  where  as  the  current 
method  says  it  is  not.  Finally,  a  possible  explanation  for  the  CMM  level  2  point  is  that 
the  model  has  a  prediction  level  of  90%,  meaning  that  it  is  possible  for  contractors  whose 
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performance  is  unacceptable,  to  fall  in  the  acceptable  range  of  the  model  1  out  of  every 
10  measurements. 
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5.  Conclusions/Recommendations 


5.1  Overview 

The  first  goal  of  this  research  was  to  establish  a  model,  based  on  the  CMM  rating 
level  of  DOD  contractors,  to  be  used  for  the  monitoring  of  contractor  performance  in 
developing  software.  The  second  goal  of  this  research  was  to  determine  the  usefulness  of 
the  above  model  to  the  acquisition  manager  in  monitoring  performance  of  contractors  on 
software  development  contracts. 

Often  acquisition  managers  use  performance  measures  for  contractors  in  different 
ways.  One  way  in  which  they  are  used  is  to  indicate  when  performance  is  below  a  set 
standard,  such  as  the  arbitrary  ±10%  limit  used  in  this  study.  This  limit  may  change 
depending  on  the  importance  or  suspected  risk  of  a  project.  A  project  that  is  very 
important  or  vital  to  an  organization  may  impose  a  limit  of  ±5%.  A  project  that  is  less 
important  might  relax  the  limit  to  ±15%.  The  results  of  this  study  suggest  that  such  a 
model  might  be  useful  in  predicting  or  monitoring  performance  of  a  software 
development  contractor  when  the  acquisition  manager  wants  to  know  if  the  contractor  is 
performing  up  to  its  capability.  This  model  can  be  used  in  conjunction  with  the  practice 
of  setting  variance  limits  on  the  contractor,  to  fulfill  multiple  monitoring  and  controlling 
functions.  In  the  following  paragraphs,  the  implications  of  this  research  will  be  explored. 
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5.2  Implications  for  the  Acquisition  Manager 

The  theory  behind  the  CMM  suggests  that  the  rating  level  of  a  contractor  can  be 
used  as  some  indication  of  the  performance  capability  of  that  contractor.  The  results 
suggest  that  the  proposed  model  in  this  study  might  prove  to  be  a  useful  tool  to  the 
acquisition  manager.  Based  on  the  model  results,  performance  can  be  predicted,  given  a 
contractors  CMM  rating  level.  Also,  the  model  can  be  used  to  determine  if  the 
performance  of  a  contractor  is  typical  of  an  organization  with  the  same  rating  level. 
However,  the  model  does  not  perform  equally  well  at  all  levels.  For  organizations  at 
CMM  level  1,  the  performance  of  the  model  is  not  good.  It  appears  that  because  CMM 
level  1  organization  performance  has  such  a  high  variance,  the  interval  in  the  model  does 
not  do  a  good  job  discriminating  between  acceptable  and  unacceptable  performance. 
Almost  any  performance  is  considered  acceptable.  This  is  important  to  the  acquisition 
manager  because  almost  70%  of  organizations  are  still  at  CMM  level  1.  As  the  rating 
level  reaches  the  higher  CMM  levels,  2  and  3,  the  model  discriminates  as  well  as  the 
arbitrary  ±10%  limit  method.  Although  this  study  did  not  contain  any  data  for  the  higher 
levels  of  the  CMM,  the  results  suggest  that  the  model  might  discriminate  at  a  level  even 
higher  than  ±10%.  As  more  and  more  companies  move  up  the  CMM  rating  scale,  the 
usefulness  of  the  proposed  model  should  increase.  The  results  are  interesting  and  suggest 
that  further  research  is  warranted  to  determine  the  full  usefulness  of  a  model  such  as  the 
one  developed  in  this  study. 
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5.3  Implications  for  the  Researcher 


CMM  theory  is  grounded  in  the  premise  that  as  CMM  rating  level  increases, 
performance  also  increases  and  becomes  more  predictable.  Correlation  studies  have 
supported  the  performance  aspect  of  this  premise.  A  natural  extension  to  this  premise  is 
that,  given  data  for  organizations  at  the  different  levels,  a  model  can  be  developed  which 
could  be  used  to  predict  performance  at  each  level.  The  results  of  this  study  support  this 
extension  for  the  higher  levels,  2  and  3,  of  the  CMM.  However,  it  is  interesting  that  the 
intervals  with  a  meaningful  level  of  prediction,  for  CMM  level  1  organizations,  are  so 
wide  that  no  accurate  prediction  could  be  made  with  them.  This  may  be  due  to  the 
limitations  of  this  research,  but  these  intervals  suggest  that  perhaps  the  variance  of 
organizations  at  CMM  level  1  is  so  large  that  meaningful  prediction  of  these 
organizations  is  not  possible.  Further  research  into  this  area  is  needed  to  determine  the 
predictive  ability  of  such  a  model  for  CMM  level  1  organizations. 

5.4  Limitations  of  the  Research 

There  are  two  major  areas  of  limitations  to  the  applicability  of  this  research.  The 
first  of  these  areas  is  bias  in  the  database,  the  second  is  the  content  of  the  database  itself. 
The  following  paragraphs  will  describe  in  more  detail  these  limitations. 

The  database  used  in  this  research  consists  of  second  hand  historical  data 

collected  by  a  third  party.  Because  of  this,  it  inherently  contains  bias.  The  method  of 

reporting  used  by  the  contractors  was  controlled  by  guidelines  (C/SCSC),  which  helped 

to  reduce  the  level  of  bias  introduced.  The  person  who  collected  the  data  and  constructed 

the  database  was  contacted  and  questioned  as  to  the  thought  processes  and  procedures 
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used  in  constructing  the  database.  This  was  done  in  attempt  to  identify  and  reduce  any 
bias  that  may  be  present.  Unfortunately,  it  is  impossible  to  eliminate  the  bias  completely 
or  to  fully  understand  the  nature  of  the  remaining  bias.  For  the  above  reason,  the  amount 
of  bias  present  and  the  effects  caused  by  its  presence  are  unknown. 

There  are  several  limitations  of  the  content  of  the  database  itself  The  first  of 
these  is  breadth.  Data  for  the  database  was  collected  from  DOD  contractors  who  had 
reported  data  to  ASC  and  ESC  as  part  of  their  contracts.  ASC  and  ESC  contractors  do 
not  represent  the  full  range  of  contractors  providing  software  to  the  DOD.  Information 
from  contractors  performing  work  for  SMC  would  greatly  add  to  the  breadth  of  the 
database,  but  unfortunately  SMC  does  not  maintain  the  information  in  a  format 
compatible  for  use  in  the  database.  Another  limitation  of  the  database  is  size.  Even  if  the 
content  of  the  database  sufficiently  covered  the  full  range  of  contractors,  it  would  still 
contain  only  a  small  sample.  The  small  number  of  data  points  available  for  model 
development  and  model  validation  affected  the  sensitivity  of  the  samples  when  data  was 
removed  for  use  in  model  validation.  This  sensitivity  might  have  caused  of  some  values 
obtained  to  deviate  from  theoretical  expectations.  The  value  for  means  and  the  width  of 
the  prediction  intervals  in  the  model  may  have  been  affected.  A  much  larger  database 
would  allow  the  development  of  a  more  accurate  model  and  possibly  more  useful  model. 

5.5  Recommendations 

There  are  several  areas  of  opportunities  for  further  research  based  on  some  of  the 
limitations  and  the  results  of  this  research.  Recommendations  for  further  research  are 


described  in  the  following  paragraphs. 


One  recommendation  is  to  broaden  the  database  and  revise  the  model.  There  are 


several  ways  in  which  the  database  can  be  broadened  that  would  lend  to  a  more  accurate 
and  useful  model.  The  first  of  these  is  to  add  data  for  contractors  with  level  4  and  level  5 
maturity  ratings.  At  the  time  the  database  was  constructed  there  were  very  few 
contractors  at  these  higher  levels.  Although  there  are  still  not  many,  there  may  be  enough 
to  develop  a  preliminary  model  at  these  levels.  A  second  area  in  which  the  database  can 
be  broadened  is  the  addition  of  space  systems.  At  this  time  the  SMC  database  is  not  in  a 
format  that  could  be  used  for  this  research.  Collection  of  relevant  information  on  SMC 
contractors  would  extend  the  range  of  the  database  and  add  to  its  validity.  Finally,  more 
data  points  could  be  added  at  the  lower  levels.  This  preliminary  study  had  a  small  sample 
set  from  which  to  develop  the  model.  Additional  data  at  the  lower  levels  would  help  in 
developing  a  more  accurate  and  possibly  more  useful  model. 

Another  recommendation  for  further  research  is  to  revalidate  the  new  model  with 
a  different  set  of  data.  There  are  two  ways  in  which  this  could  be  accomplished.  The 
first  way  is  to  validate  the  model  with  more  points  from  a  single  contract.  In  this  study 
the  model  was  validated  using  a  single  point  from  different  contracts.  Although  this  was 
sufficient  for  this  preliminary  study,  validation  of  the  model  using  multiple  points  from  a 
single  project  might  be  of  value  in  determining  the  usefulness  of  the  model  over  time. 

The  second  way  to  accomplish  revalidation  is  to  attempt  to  revalidate  using  a  much  larger 
sample  size  at  each  of  the  rating  levels.  For  this  preliminary  study  only  three  data  points 
were  used  at  each  of  the  first  three  CMM  rating  levels.  Increasing  the  number  of  points 
used  would  allow  the  researcher  to  validate  the  prediction  level  proposed  for  the  model 
while  validating  its  usefulness  to  the  acquisition  manager. 
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5.6  Conclusion 


The  goal  of  this  preliminary  study  was  to  evaluate  the  possibility  of  creating  a 
model  based  on  the  CMM  rating  level  of  contractors  and  to  determine  the  usefulness  of 
such  a  model  to  the  acquisition  manager.  The  results  of  this  study  suggest  that  such  a 
model  might  be  possible  and  useful  as  a  tool  to  monitor  and  control  contractor 
performance,  and  that  further  research  in  this  area  is  warranted. 
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Appendix  A:  Unreduced  Data  Set 


This  appendix  provides  the  unreduced  data  set  contained  in  a  Microsoft  Access 
version  2.0  database.  Each  database  record  representing  an  individual  data  point  is 
presented  in  a  “form”  format,  with  each  record  represented  by  a  separate  page. 


A-1 


Data  Identification 


OrgTag:'  ,  RatingTag:  [a^  V\®S#;  |T 

WBSDescription: 


iOperational  mission  software  planning,  requirememts  analysis,  change  review/assessment  review/approval  requirements 


Rating  Information 

Rating  Date: 


10/15/93  Rating: 


Rating  Type; 


Moderating  Variables 


Acquisition  Phase:  ISupport/Upgrade 


Contract  Type: 


Program  Comments 


SfW  Lifecycle:  I Req u irements 


100.00%  Application:  I  Avionics 


16608000  Budget  VolaSlity:  ILow 


156600 


%  New/Modl^ed  Code:  I  100.00% 


Requirements  Volatility:  lUnk 


QuaH^  Stds  On  Contract:  Quality  Params  Tracked : 


Cost  Accoun^ng  Anomalies:  iVarian  ces  may  be  influenced  by  letter  contract  prior  to  periods  of  interest 


Program  Bfenager  Comments:  ISize  was  converted 


Six  Months  Prior  to 
Rahng 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  5/30/93 


8/30/93 


1/30/94 


4/30/94 


8CWS: 


BCm>: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP; 


Budget: 


16782 


16608 


16168 


16698 


Budget  Volatility  Index: 


LRE  Volatiilty  Index:  j  0.0416 


Percent  Complete:  I  0.2888 


BCWSAcdvity:  f  0,32902 


BCWPActIvify:  j  0.43402 


ACWPAcdv%:  j  0.4667 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index:  I  0-58075 


Invesh'gator  Comments: 


Data  Identification 


WBSDescnptlon:  planning  and  integration  of  operational  nnission  software 


Rating  Information) 

Rating  Date:  |  10/15/93 


Rating  Type:  |SPA(EXT) 


Moderating  Variables 


Acquisition  Phase:  iSupport/Upgrade 


Program  Comments: 


S/W  Lifecycle:  |l  nteg  ration 


Language  %:  [100.00%  Application:  lAvionics 


5186000  Budget  VolatiUly:  iLow 


%  New/Moditied  Code: 


Requirements  Volatility:  lUnk 


Quality  Stds  On  Contract  ^  Quality  Params  Tracked : 


Rebasellntng 


Cost  Accounting  Anomalies:  1  Variances  may  be  influenced  by  letter  contract  prior  to  periods  of  interest-check  for  rebaselining 


[sews  decreased 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


8/30/98 


1/30/94 


4/30/94 


BCWP: 


8CWP: 


BCWP; 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


5902  Budget: 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index; 


BCWSActivi^:  I  0,47671 


BCWP  Activity:  I  0.57766 


ACWP  Activity:  I  0.55111 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index:  I  ■) -70968 


Data  Identification 


WBSpescnptiqn:  ^  iPianning,  design,  implementation  and  test  of  operating  system 


Rating  Information 

Rating  Date:  I  10/15/93 


Rating  Type;  |SPA(EXT) 


RateComment: 


Moderating  Variables 


Acquisition  Phase:  iSupport/Upgrade 


Contract  Type:  iCPl 


Program  Comments: 


SW  Lifecycle:  ■■  |y ultipte 


Language  %:  |  87.00%  Application:  lAvlonics 


4201 000  Budget  Volatility: 


%  Nevif/Modified  Code: 


16300 


Requirements  VoladHty:  [Unk 


Quality  Stds  On  Contract:  bf:  Quality  Params  Tracked : 


Gc^  Accoundng  Anomalies::  jVariances  niay  be  influenced  by  letter  contract  prior  to  periods  of  interesi 


Six  Months  Prior  to 
;  l^dng 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Raring 


Six  Months  After 
Raring 


Da^:  I  5/30/93 


8/30/93 


1/30/94 


4/30/94 


BCWP: 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


5355  Budget: 


Derived  Moderators 


Budget  Vol^lity  index: 


LRE  Volarility  Index:  I  43.209 


SGWSAcrivlty:.  j  0.46004 


BCWPAcrivtty:  I  0.55908 


ACWPActhrity:  j  0  46634 


Dependent  Variables 


Schedule  Performance  index: 


Data  identification 


WBSDescHptfon:  ^  lAnalyze.  design,  and  code  software  for  sofhvare  simulation  system  component 


Rating  infonmation 

RaHng  Date;  |  1/15/94  Rating; 

RateComment 


Moderating  Variables 


Acquisl^on  Phase:  lEMD 


Program  Comments:  jiVlay  have  incentive  fee  on  contract-did  not  show  up  in  CPR 


S/W  Lifecycle:  [Code/T est 


Language  %:  i  1 00.00%  Application:  ISimulation 


4300000  Budget  VolatilHy:  llVled 


%  Hew/Modified  Code:  1  100.00% 


R^ulrements  Volatility:  |High  .  Rebaseltning :  jNo  Quality  Stds  On  Contract:  Quality  Params  Tracked : 

CostAccounting  Anomall^:  ■ iRebaselfning  pnor  to  toeframe  of  interest.  May  see  reperaisstons.  ' : = ■ 


Program  Manager  Comments::  /  iThe  government  may  be  responsible  for  50%  of  problems  ie  cost/schedule  variances.  Contractor  has 


done  a  ’’competent  job’ 


Six  Months  Pnorto 
V"  Radng  " 


Three  Months  Prior  to 
R^ng 


Three  Months 
After  Rating 


Six  Montihs  After 
'  Rating 


Date:'  I  11/30/93 


3/30/94 


7/30/94 


BCWS: 


BCWP; 


ACWP: 


ACWP: 


ACWP; 


ACWP: 


DeHved  Moderators 


BudgetVolatHi^Index;  I  0.53188  LRE  Volatility  Index:  f  0.5217 


Percent  Complete:  I  0.8751 


BCWS  Activity:  C.26576 


BCWP  Activity:  I  0.25751 


ACWP  Activity:  jO. 24861 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index:  i  1 -03526 


Data  Identificatioh 

OrgTag:  [I'  RatingTag:  jl'  V\®S#:  p" 
WBSD^cripticn: 


Analyze,  design,  and  code  software  for  software  simulation  system  component 


Rating  information 

Rating  Date:  I  1/15/94 


Rating  Type:  |SPA(EXT) 


Moderating  Variables 


Acquisition  Phase: 
Program  Comments: 


S/W  Lifecycle:  ICode/Tesi 


Language  %; 


100,00%  .  :  Application:  |Simulation 


334 1 000  Budget  Volatility: 


22712  :  ;  %  New/Modified  Code:  i  100.00% 


Requirements  Volatility:  I  High 


Quality  Stds  On  Contract:  h/!  Quality  Params  Tracked : 


Cost  Ac^unting  Anomalies:  iRebaselinlng  occurrred  immediately  prior  to  timeframe  of  interest  May  see  repercussions. 


Program  Hfeinager  Comments:  -;:jThe  government  may  be  responsible  for  50%  of  the  problems  ie  cost/schedule  variances.  Contractor  has 


done  a  “competent  job’ 


Six  Months  Prior  to 
Rating 


Thr^  Months  Prior  to 
Radng 


Three  Months 
After  Ratifig 


Six  Months  After 
Rating 


Date:  I  8/30/93 


1/3D/9: 


3/30/94 


7/30/94 


BCWS: 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  Volatility  Index:  j  0.31847  LRE  Voladlity  Index:  f  0.2957 


Percent  Complete:  I  0.9877 


BCWS  Activity:  |  0.23773 


BCWP  Activity:  0.2435* 


AGWPAcdvjty:  I  0.21993 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index: 


Investigator  Comments: 


Data  Identification 


RaShgTag:  IB  -  WBS'#:  1$ 


WBSDescnptilon; 


Analyze,  design,  and  code  software  for  software  simulation  system  component 


Rating  Information 

;  Rating  Date:  I  1/15/94  Rating; 


RateComment 


Moderating  Variables 


SWLife«qrde:  |Cocie/Test_^_  Language:  |Ada  ■  Language'/.:  j  100.00%  Application:  jSimulation 

Project  Budget:  j  ^2365000  Budget  Volatility:  Size:  |  138837  %  New/Modified  Code:  |  100.00% 

Requirements  Votatiiity:  |High  ^  Rebaselining:  |no  Qualify  Stds  On  Contract:  Qualify  Params  Tracked : 

Cost  Accounting  Anomalies:  :  jRebaselining  oocorrred  immediately  prior  to  timeframe  of  interest.  May  see  repercussions.  :  ■: ; 


Program  Manager  Comments:,/  The  government  may  be  responsible  for  50%  of  the  problems  ie  cost/schedule  variances.  Contractor  has 
•  ■->■■■■  w-isv  .  done  a  "competent job". 


Cost  Date 

Six  Months  #r!or  to 
^ ,  Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
/  Haring 


8/30/93 


Date:  I  11/30/93 


3/30/94 


7/30/94 


BCWS: 


BCWP: 


ACWP: 


ACWP: 


ACWP; 


Derived  Moderators 


Budget  Volatility  Index;  |  0.15761  LRE  Volatitify  Index:  [  0.1484  Percent  Complete:  |  5-?784 

BCWSActivify:  |  0.11922  '  SCWPActivify:  |  0.12014  '  ACWP  Activity:  |  0.11211 


Schedule  Performance  Index: 


1.007245 


Cost  Performance  Index: 


1.06107 


invesrigator  Gommersts: 


Data  identification 


RafingTag:  |b'  WBS#:  |4 

[Analyze,  design,  and  code  software  for  software  simulation  system  componenf 


Rating  information 

Rating  Date:  |  1/15/94  Rating; 

RateComment 


Rating  Type:  ISPA  (EXT) 


Rating  Relevance:  I  High 


Moderating  Variables 


Acquisition  Phase:  lEMD 


Contract  Type: 


ICPAF 


Program  Comments:  ilVlay  have  incentive  fee  on  contract-dld  not  show  up  in  CPR 


S/W  Lifecycle:  ICode/Test 


Language  %:  1100.00%  Application:  jSimufatior 


Project  Budget: 


8685000  Budget  Volahiity :  (High 


%  New/Modified  Code:  i  100.00% 


R^ujrementsVolatiiity:  (^High  Refa^eiming  :  pes  Quality  Stds  On  Contract  Qualify  Pamms  Tracked : 

Cost  Accounting  Anomalies: ;  j  in  Sep  94,  a  reallocation  of  budget  was  detected.  Prior  to  thlsT  they  were  on  budg^^^  and  on  schedule 


Program  Manager  Comments:  The  government  may  be  responsible  for  50%  of  the  problems  ie  cost/schedule  variances.  Contractor  has 
^ '  done  a  '‘competent  job”. 


Six  Months  Prior  to 
Rating 


Three  Months  Pnorto 
Rating 


Three  Mondis 
After  Rating 


Six  Months  After 
Rating 


Date:  i  8/30/93 


Date:  I  11/30/93 


8CWP: 


BCWP: 


BCWP; 


ACWF: 


ACWP: 


ACWP: 


ACWP: 


Budget 


1378  Budget 


10085 


Derived  Moderators 


Budget  Volatitity  Index:  |  5.30261  LRE  Volatility  Index:  j  6.2398  Percent  Complete:  |  0.2251 

BCWS  Activity:  |  0.16024  BCWP  Activity:  j  0.1509  ACWP  Activity:  j  0.14403' 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index: 


Data  identification  ^ 

OrgTag:  ftofingTag:  JT  ;  WBS#:  |i 

WBSDescripfion: ::  joesign,  code,  an<j  test  flight  control  software 


Rating  information 

Rating  Date:  - 1  5/15/92  luting: 

RateComment 


Rating  Type:  I  SPA  (EXT) 


Rating  Relevance:  [High 


Moderating  Variabies 


Acquisition  Phase:  I  Production 


S/W  Lifecycle:  jReiease  ^  Language:  |Joviai  ,  Unguage%:  1 100,00%  Application:  |Avionics 

Project  Budget:  |  __  3622000  Budget  Volatility:  jNone^  Size;  j  %  New/Modified  Code;  J  '*00.00% 

RequirenjentsVotefility:  |low  Rebaselining:  |no  Quality  StdS On  Contract:  L.  Quality  Params Tracked : 

Cost  Accounting  Aiwmall^: .  :  |Mtnimal  effort-Largely  complete.  May  not  be  enough  effort  to  be  a  valid  data  point  ^ 


Additional  requirements  &  darifications  determined  to  be  in  or  out  of  scope.  Out-of-scope 
added  as  ECPs 


Six  Months  Pnortci 
Rating  * 


Thi^  Months  Pnor  to 
Rating 


Three  Monte 
After  Rating 


Six  Monte  Afer 
Rating 


7/30/92 


Date:  I  11/'30/92; 


BCWP; 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  Volatility  Index:  |  0.00194  ;/  LRE  Volatility  Index; 

BCWSActi>dty:  j  0.0017  ,  ,  BCWP  Activity:  |  0.00169 


PercentComplete:  •  i  0.9787 


ACWP  Activity:  I  0.00296 


Dependent  Variables 


Schedule  Performance  index; 


Cost  Performance  Index:  I  0.54545 


Data  point  exduded  from  Complete  Data  Set  due  to  low  activity  level 


Data  identlRcation 


OrgTag:  p'  RatingTag:  pT*  WBS#:  j2 

W^Desciipfion:  joefine  requirements  for  each  CSCI,  perform  updates  to  legacy  system 


Rating  information 

,  Rafing  Date:  [  S/15/91  ■  Rating: 


RateCommeirt:  I  information  provided  by  Contractor  (no  program  office  intermediary) 


Moderating  Variables 


Acquisi^on  Phase:  jEI\^D 


ContractType:  ICPFl 


Program  Comments:  iProgram  was  cancelled 


SiWIJfecycfe:  iTest/Integration  Language; 


00.00%  : .  Appiication:  lothe: 


Project  Budget:  |  6282000  Budget  Voiatility 

Requirements  Volatiirty:  :-:::[High  Rebaselinin 


%  Mew/Modified  Code: 


150000 


Ouality  Stds  On  Contract  ^  Quality  Params  Tracked 


Cost  Accounting  Anomalies; 


Program  Manager  Comments: :  iProgram  was  "overcome  by  events”  and  was  thus  cancelled. 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:!  12/30/90 


3/30/91 


Date:  I  11/30/9 


BCWS 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget: 


4850  Budget 


Derived  Moderators 


Budget  Volatility  Index:  j  0.41 327 


LRE  Volatilify  Index:  I  0.41 1 1 


Percent  Complete:  \  0.7954 


BCWSAct\nty:  0.25171 


BCWP  Activity:  |  0.27176 


ACWP  Activity:  0.25862 


Schedule  Performance  Index: 


Cost  Performance  index; 


Data  identlflcallon 


VVBSDescriptio'n:  iDesign,  code.  test,  and  integration  of  sofb.vare  for  flight  control  system 


Rating  Information 

Rating  Date:  |  5/15/92  Rating: 

RateComment: 


Moderating  Variables 


Acquisition  Phase: 


Contract  Type: 


Program  Comments:  r’Cost  plus  some  base  fee  plus  any  incentive  (sic)  fees  awarded’ 


SW  Lifecycle: :  j Multiple-Early 


Language  %:  1100.00%  Application:  lAvionrcs 


Project  Budget:  |  316251000  Budget  Volatility :  jLow 

Requirements  Volatility:  jtow  Rebaselining  :  p3c 


%  New/Modified  Code:  I  100.00% 


70000 


Quality  Stds  On  Contract:  Quality  Params  Tracked :  ^ 


Program  Manager  Comments:  .  I Personnel  highly  experienced  in  digital  flight  control  systems 


Six  Months  Piior  to 
Rating 


Three  Months  Prior  to 
,  starting 


Three  Monte 
After  Rating 


Six  Monte  After 
Rating 


3/30/S2 


8/30/92 


Date:  I  11/30/92 


BCWP: 


741$ 


BCWP; 


BCWP: 


ACWP: 


ACWP; 


ACWP: 


19140 


ACWP: 


316251 


316251 


Derived  Moderators 


Budget  Vbla^lity  Index: 


LRE  Volatilily  Index: 


BCVVSAcdvity: 


BCWP  Activity 


ACWP  Activity: 


Dependent  Variables 

Schedule  Peifonnance  Index: 


Cost  Performance  Index:  I  o  92732 


Data  Identification 

OfQTsg;  J^^n9T39-  F  WBS#:  ^ 

WBSDescripfcion: 


Design,  code,  test,  and  integration  of  low-level  hardware/software  routines  for  diem 


Rating  Information 

Rating  Date:  i  5/15/92 


Rating  Relevance:  iMed 


Moderating  Variables 


Acquisition  Phase:  lEUD 


Program  Comments: 


S/W  lifecycle:  jMuSpie-Eariy  ^inguage:  |Ada 

Project  Budget:  45545^0  Budget  Volatility : 

Requirements  Vola^Hty:  (ivied  F^baselining 


Language  %:  j  75.00%  Application:  lAvionics 


15000 


%  New/Modrfied  Code:  j  100.00% 


Quality  Stds  On  Contract:  Quality  Params  Tracked : 


Cost  Accounting  Anomalies; 


Six  Months  Prior  to 
Radng 


Three  Montt^  Prior  to 
Rating 


Three  Months 
After  Rahng 


Six  Months  After 
Rating 


Date:  I  12/30/9 


3/30/92 


8/30/92 


Date:  I  11/30/92 


BCWS: 


BCWS: 


BCWS: 


BCWP: 


BCWP; 


ACWP; 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  VqlaSli^  Index:  |___  j0.0635^  .  LRE  Volatility  index: 
BCWSActivffy:  .1  1  .  BCWPActivi^:  j  1 


ACWP  Activity: 


Dependent  Variables 


Schedule  Performance  Index; 


Cost  Performance  Index:  I  0.94571 


Data  Identification  : 

W  WBS  #:  |T 

WBSDescnplaon: 


: Design,  code,  test,  and  integration  of  software  for  flight  control  system 


Rating  information 

Rating  Date:  |  10/15/93  Rating: 

RateComment 


Rating  Type:  iSPA  (EXT) 


Moderating  Variables 

Acquisition  Phase:  |emd 
Program  Comments:  f  ~  '  "  ' 


S/W Lifecycle:  |Multiple^  Language:  |Ada_  ^  Language  %:  j[l 00.00%  AppllcalSon:  jAvionics 

Project  Budget:  j  262222000  Budget  Volatility:  jHsgh  ^  ^  Size:  j  ^Toooo*  %  New/Modified  Code:  |  100.00% 

Requirements  Volatility:  |low  Rebas^lning :  Jves  ^  Quality  Stds  On  Contract:  ^  Quality  Params  Tracked :  3^ 

Cost  Accoundng  Anomalies:  jRephased  during  this  period-some  aberrations  may  be  attributable  to  the  contractor  anticipating  the  coming 
^  ,  [rephases  and  delaying  expenditures 


Program  Manager  Comments:  .  iPerscnnel  highly  experienced  in  digital  flight  controi  systems 


Sa  Months  Prior  to 
Rating 


Three  Months  Prior  hi  / ; , 
Rating 

I  - 

Date:  |  8/30/93 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  i  5/30/93 


VZO/9^ 


4/30/94 


46194 


79080 


BCWP: 


BCWP; 


ACWP: 


ACWP; 


ACWP: 


64021 


ACWP: 


300751 


300751 


Derived  Moderators 


Budget  Vplahlity  Index: 


^  LRE  Voladlify  Index:  |  ^  ^167  Percent  Comf^ete:  |  0.2953 

BCWP  Activity:  |  0.43588  .  ACWAcb'^:  t  0.40598 


BCVVSActwl^:  I  0.41586 


Dependent  Variables 


Schedule  Performance  Index: 


1.026181 


Cost  Performance  lndex^  i  0271? 


Note  decrease  in  Budget  and  LRE  dunng  this  12  month  period. 


Data  Identification 


OrgTag:  .  RatingTag:  |s*^  WBS#:  |2 

. WBSDescnptfon^  ■...  I^Design,  code,  test  and  integration  of  low-leve!  hardware/software  routines  for  client 


Rating  Information 

'  Rating  Date:  |  10/15/93  Rating: 

"  RateComment 


Moderating  Variables 


Acquisition  Rtase:  lElVlD 


Program  Comments: 


S/W  Lifecycle: 


Language  %:  I  75.00%  Application:  lAvionics 


$7704000  :  Budget  Volatility :  IHigh 


%  New/Modified  Code: :  f  100.00% 


15000 


Requirements  Voladii^:  pled  Rebaselining:  |Yes  Ouality  Stds  On  Contract:  .V;  Quality  Params  Tracked : 

Cost  Accounting  Anomalies:  Rephased  during  this  penod-some  aberrations  may  be  attributable  to  the  contractor  anticipating  the  coming 
\  rephases  and  delaying  expenditures 


Program  Manager  Comments:  I  Personnel  highly  experienced  in  digital  flight  control  systems 


Six  Months  Prior  to 
X  Radhg, 


Three  Mondis  Prior  to 
Rating 


Three  Months 
After  Raring 


Six  Months  After 
Rating 


Date:  5/30/93 


8/30/93 


1/30/94 


4/30/94 


33814 


BCWP: 


BCWP: 


BCWP: 


BCWP: 


38160 


ACWP: 


ACWP: 


ACWP: 


61045  Budget: 


0  Budget: 


S7704 


Budget  Volarility  Index:  j  0.43671 


LRE  VolariJity  Index:  j  0.4561 


Percent  Complete:  I  0.4351 


BCWS  Acriv^:  I  0.47089 


BCWP  Activity:  j  0.47476 


ACWP  Acrivity :  j  0.461 55 


Dependent  Variables 


Schedule  Petfomiance  Index: 


Cost  Performance  index:  |  0.93222 


Data  tdentificatioii 


OigTagf  'p';:.-' 

WBSDescrfption:  |De$ign,  develop,  code,  test,  and  install  2  Flight  Programs,  2  Ground  Programs 


Rating  Information 

RalSng  Date:  |  11/15/92 


Moderating  Variables 

Acquisition  Phase:  \£MO 


Contract  Type:  IFPiF 


Language:  lAda 


Applica^on:  lAvionics 


12457000  Budget  Volafility:  iLow 


1S0000 


%  New/Modified  Code:  f  100.00% 


Requirements  Voi^lity:  |Low 


Rebasellning :  I  No 


Quality  Stds  On  Contract: 


Quality  Params  Tracked 


Cost  Accounting  Anomalies:  |No  revised  over-target  baseline  since  1 989 


Program  Manager  Comments:  jSoftware  is  in  the  "top  10"  budget  drivers,  and  is  a  key  issue  on  the  program.  Subsystems  well  defined,  but 

jthere  have  been  integration  challenges. 


Six  Months  Prior  to 
Rating" 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


S  ix  Months  After 
Ra^ng 


Date:  f  5/30/92 


8/30/92 


1/30/9; 


4/30/93 


BCWS: 


BCWS: 


10387 


BCWP: 


10983 


10598 


BCWP: 


ACWP: 


ACWP: 


17431 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  Vol^lity  Index:  |  0.10916  LRE  Volatility  Index:  ^  0.1031  PercerrtCo 

BCWSAcSvl^;  I  0.05256,'  BCWPActivi^:  |  0.05427  ACWP  Activity:  I  0.0835 


Dependent  Variables 


Schedule  Performancei  index 


Cost  Performance  Index:  I  C-38o26 


Selected  for  model  validation. 


Rating  Information 

Ra^ng  Date:  |  12/15/90  '  Rating: 

RateComment 


Rating  Type:  |SPA(ir'rr) 


Rating  Relevance:  jVery  High 


Moderating  Variables 


Acquisi^on  Phase:  lEUD 


i  Fortran 


Language  %:  I  61.00%  ■  >  Application:  I  Command  &  Co 


22788000  BudgetVolatility:  iLow 


430000 


%  New/Wodified  Code: 


Requirements  Volahlity:  jHrgh  Rebaselining  :  jNo  Quality  Stds  On  Contract,  y!  Quality  Params  Tracked  :  y 

Cost  Accounting  Anomalies:  jstop  woi1<  orders,  change  in  direction,  etc  rt^y  affect  performance  Indices  '  ”  " '  '  ' 


Program  Manager. Comments: iThinks  contractor  is  a  level  2.  '’Contractor  is  not  as  good  as  some,  but  better  than  most' 


Six  Months  Pnor  to 
Rating 


Three  Months  Poor  to 
Rahng 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


9/30/90 


5/30/91 


BCWS: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget: 


78000 


Derived  Moderators 


Budget  VplafiHty  Index:  f  0.00057  LRE  Volafilify  Index; 


Percent  Complete:  I  0.9939 


BCWS  Activity:  I  0.05207 


BCWP  Activity:  I  0.0978 


ACWP  Activity:  |  0.15214 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index:  |  0.20188 


CPi  moderate  tow  outHen  SP!  extreme  high  outlier.  Investigation  reveals  valid  data  point.  Negligible  influence  on  cumulative  SPI 
K.94S  to  .994).  and  CPI  (.334  to  .314). 


A-i6 


Data  Identification 

Oi^Tag:  jc'  • 
WBSD^cnp^on: 


f^tingTag:  [s'  WBS#: 

[software  engineering  efforts  to  define,  develop,  and  test  system  software 


Rating  Information 

Rating  Date:  |  11/15/92  Rating: 

RateComment: 


Rating  Relevance:-  ;- jVery  High 


Moderating  Variables 


SM  lifecycle:  (Multiple 


[Fortran 


Language  %:  I  61 .00%  Application:  jCommand  &  Co 


Project  Budget; 


82378000  Budget  Volatility:  (Low 


%  New/Modified  Code: 


430000 


Quality  Stds  On  Contract:  Quality  Params  Tracked :  yt 


Requirements  Volatility:  (High 


Rebaselining :  (No 


Cost  Accounting  Anomalies:  ;  (Stop  work  orders,  change  in  direction,  etc  may  affect  performance  irtdices 


Program  fyianager^Comments;-;/ |Th^^  contractor  is  a  level  2  "Contractor  is  not  as  good  as  some,  better  than  most" 


Six  Mondis  Prior  to 
Radng 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  !  5/30/92 


9/30/92 


4/30/93 


BCWS: 


SCWP; 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


81S95 


Budget 


LRE:  I  82692 

Derived  Moderators 


85431 


Budget  Volatility  Index: 


LRE  Volatility  Index:  (  0.0456 


BCWS  Activity:  j  0.0127 


BCWPActi>^:  (  0.01368 


ACWP  Activity:  (  0.03769 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Perforinance  Index:  1  0.34957 


Seiected  for  mode!  validation. 


A-17 


Data  identification 

OrgTag:  |iT  RatingTag:  V\®S#:  [T 

WBSDescnption: 


jOesign,  code,  test,  and  integration  of  software  CPCis 


Rating  Information 

Rating  Date:  j  11/15/92  Rating: 


Rating  Relevance:  [Very  High 


Moderating  Variables 


Acquisition  Phase:  lEMD 


ContractType: 


Program  Comments: 


S/W  Lifecycle:  iMuitiple 


93.00%  Application:  (Command  &  Co 


Project  Budget: 


12860000  Budget  Vola^lity: 


%  New/Modifled  Code: 


357714 


Requirements  Volatility:  (Med 


Quality  Stds  On  Contract:  Quality  Params  Tracked : 


Cost  Accourtdng  Anomalies: 


linternat  reallocated  effort- 


program  Manager  Comments; 


Highly  concurrent  effort.  ECPs  effectively  doubled  scope  of  the  effort  without  stretching  schedule-thus 
increased  program  schedule  risk.  Program  Manager  thinks  contractor  is  level  2.  "'Not  as  good  as  some,  but 
better  than  most”. 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Radng 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  5/30/32 


1/30/93 


4/30/93 


BCWP: 


BCWP: 


BCWP: 


BCW>: 


ACWP: 


ACWP: 


ACWP; 


ACWP: 


Budget 


16421 


12860: 


Derived  Moderator 


Budget  Volatility  Index: 


LRE  Volatilitylndex:  (  -0.2018 


Percent  Complete:  T  0.4762 


BCWS  Activity:  I  0.53658 


BCWPAchVify:  I  0.56695 


ACWP  Activity:  I  0.63998 


Schedule  Performance  Index: 


.04736 


Selected  for  model  validation, 


Data  Identification  :  ;  - 

OrgTag:  jT'  "RafingTag:  [iT  WBS#:  |l 

v,"W^P®scnption::j,  jsoftware-Related  managenienraSvitie^  8aselining!^oSrare^e^opment  planning,  etc 


Rating  infomiation 

Rating  Date:  I  4/15/90'  Rating; 


Rating  Type:  jSPA(EXT} 


RateComment:  jSEl  conducted  the  rating 


Moderating  Variables 


$/WUfecyde:  puitipi^Eidy  Language: 

Project  Budget;  J  Budget  VolatifHy:  |low_  _ 

Requirements Volatlity:  |low_  Rebaselining:  lYes'l 

Cost  Accounting  Anomalies:  IbCWS  decreased  in  last  6  months  of  period 


Language  %:  [  0.00%  Application:  [Database 


%  New/Modil^ea  Code: 


Quality  Stds  On  Contracfc  3^!!!!  -  Quality  Params  Tracked :  5/! 


^NOTE*"^  Quality  standard  in  this  case  is  2167  (tailofed)-need  to  determine  if  DOD-STD-2168  or  Di-QClC 
80572  is  on  contract. 


Six  Monttis  Piior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
Aft^  Rating 


Date:!  10/30/89 


1/30/90 


BCWS: 


BCWP: 


BCWP: 


ACWP: 


ACWP; 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Volatlity  Ind^:  |  0.05968  LRE  Volatility  Index:  |  0.0362  Percent  Complete:  j  0.8^7 

BCWS  Activity:  j  0,1469'  BCWPActivi^:  [  0.1469  "  ACWP  Activity:  j  0.15159 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index:  J  0.96737, 


OrgTag:  RatingTag:  jpT  WBS#:  |2 


V\BSDescnptlon:{;-  |^  design  and  integration  oversight  tasks.  Code  and  unit  test  of  database  architecture, 


Rating  Infonnatlon 

Rating  Date:  |  4/15/90  Rating: 


RatogType:  jSPA(EXT) 


RateCoinment:  jSEl  conducted  the  rating 


Moderating  Variabies 


Acqulsi^on  Phase:  I  Upgrade 


S/W  Lifecycle:  {Multiple 


Language  %:  1 100,00%  Applicahon:  {Database 


4602000  Budget  Volatility:  (tow 


%  New/Modified  Code; 


40000 


Requirements  Volatilify:  {Low 


Quality  Stds  On  Contract  y  Quality  Params  Tracked  :  y: 


Cost  Accounting  Anomalfe*  ■  ^  jRebaselining  prior  to  this  period  does  not  affect  this  measuremeni 


Program  Manager  Comments:  Quality  standard  in  this  case  is  2167  (tai!ored)-need  to  determine  if  DOD«STD-2168  or  DI-QCiO 


80572  is  on  contract 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Montos 
After  Rating 


Six  Months  After 
Rating 


Date:  I  10/30/89 


1/30/90 


6/30/90 


BCWS: 


BCWP: 


8CWP: 


BCWP: 


ACWP: 


ACWF: 


ACWP; 


ACWP: 


Budget: 


Budget 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index:  |  0.5158 


Percent  Complete:  [  0.5693 


BCWSActh^ity:  |  0.14384 


BCWP  Activity  :  0.1687 


ACWPAcdvity:  [  0.20374 


Dependent  Variables 


Schedule  Performance  index: 


1.172414 


Cost  Performance  index: 


Investigator  Comments: 


Data  identification 


OrgTag:  li 


•WBSDescription:'  ■  iSubsystem  test,  test  planning  and  integration 


Moderating  Variables 


Acqufettion  Phase: 
Program  Comments: 


SA/V  Lifecycle:  |Tes< 


Language  %:  j  0.00%  :  Appnca^on:  j Database 


14830000  Budget  Volatility:  Low 


%  New/Modified  Code; 


Requirements  Volatility:  |Low  Rebaselining  :  |No  Quality  Stds  On  Contract:  Quality  Params  Tracked : 

Cost  Accounting  Anomalies:  jRebaseHning  pnor  to  this  period  does  not  affect  this  measurement-increase  in  budget  in  later  qtr.  '  ■ 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Radng 


Three  Mondis 
After  Rating 


Six  Monttis  After 


Date:  I  10/30/89 


./30/90 


6/30/90 


9/30/90 


BCWS: 


BCWP: 


BCWP: 


BCWP; 


ACWP: 


ACWP: 


ACWP: 


ACWP; 


10374 


14880 


10900 


Derived  Moderators 


Budget  Volatility  Index:  I  0,4551 1 


LRE  Volatility  Index:  [  0.4309 


Percent  Complete; 


BCWSActhrity:  I  0,25317 


BCWP  Activity; 


ACWPAcdvity:  I  0,27827 


Schedule  Perfomiance  index: 


1.086413 


Cost  Performance  index:  I  0.98556 


Investigator  Clements: 


Data  identification 

OrgTag:  f**  /  Ratin^tag:  :  WBS#;  F“ 

WBSDesciiption:  jDesrgn,  code  and  unit  test  of  CSCIs 


Rating  Information 

Rating  Date:  I  4/15/90  Rating: 


Rating  Type:  jSPA  (EXT) 


RateGomment:  |S£i  conducted  the  rating 


Moderating  Variables 


Acquisition  Phase: 


‘Upgrade 


Contract  Type;  [PPIF 


Program  Comments: 


SAA^  Lifecycie:  iMaltiple 


Lan3uage%:  1 100.00%  Application:  | Database 


16453000  Budget  Volatiiity:  jlow 


755600 


%  New/Moditied  Code: 


Requirements  Volatility:  jLow  Rebaselining :  jNo 

Cost  Accounting  Anomalies;  |  Budget  increases  in  Sept 


Quality  Stefs  On  Contract:  3^  Quality  Params  Tracked  :  S 


Program  Manager  Comments 


Six  Months  Prior  to 
Rating 


Three  Montiis  Pne^-to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  10/30/89 


1/30/90 


BCWS: 


BCWS: 


BCWP: 


BCWP: 


7210 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


10443 


Budget: 


10512  Budget: 


10512 


Budget 


ime 


12147 


17109 


Derived  Moderators 


Budget  Volatility  index: 


LRE  Volatility  Index:  I  0.5254 


BCVyS  Activity:  I  0.29112 


BCWP  Activity:  I  0.31802 


ACWP  Activity:  I  0.31409 


Dependent  Variables 


Schedule  Perfonnance  index: 


Cost  Performance  Index:  I  0.95995 


Data  identification  ■  : 

OrgTagr  .p”  RatingTag:  j^T  WBs  ifc  F 

■^'WBSDesciiption:  [Design,  code  and  unit  test  of  CS(5s 


Rating  Information 

RatogD^:  I  4/15^0 


RateComment: 


SEl  condiicted  the  rating 


Moderating  Variables 


Acquisl^on  Phase:  jUpgrade 


Program  Comments: 


S/W  yfecycle:  [Multiple 


Language  %:  1100.00%  Application:  [Database 


3822000  Budget  Volatility:  ILow 


%  New/Modifred  Code: 


Requirements  Voladffty :  [tow 


Quality  Stds  On  Contract  yfH  Quality  Params  Tracked :  y. 


Cost  Accounting  Anomalies; 


Program  Manager  Comments: 


Three  Months  Prlorto 
Radng 


Six  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
l^'ng 


Date:  [  10/30/89 


J30/90 


6/30/90 


BCWP: 


BCWP: 


ACWP: 


ACWP; 


ACWP: 


ACWP: 


Budget: 


3077 


Budget 


Derived  Moderators 


Budget  Volatility  Index:  |  0,25055 


LRE  Volatility  Index:  |  0,2813 


Percent  Complete:  0>6164 


BCWSAcav«y:  I  0.093S9 


BCWP  Activity;  I  0.11503 


ACWPActivi^:  I  0.14541 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index: 


Data  identification 


OrgTag:  RatangTag:  f  WBS#:  [l 

WBSDescriptilon:  jsoftware-Related  management  activities:  Baselming,  Software  development  planning,  etc 


Rating  Information 

Ra^ngDate;  |  10/15/91  Rating: 

?  RateComment 


Rating  Type:  jSCE 


Moderating  Variables 


AcQuisitfon  Phase:  [Upgrade 


Contract  Type: 


Program  Comments:  .  [contract  converted  from  FPl  to  FPf/CPFF  during  this  penod 


S/W  Lifecycle: :  -I  Multiple-Early 


[Other 


0.00%  .  Application:  [Database 


Project  Budget: 


2521 DOO  -  .  Budget  Voiatirity:  JLow 


%  New/Modified  Code: 


Requirements  Volatility:  [Med 


Quality  Params  Tracked 


Cost  Accounting  Anomalies:  ^"INVALID  DATA  POINT^^May  have  moved  work  during  this  period  (Aug  91)-indicated  decrease  in  budget  and 

actuals 


Program  Manager  Comments: 


Cost  Data 

Six  Months  Pilorto 
Rating 


Three  Mondis  Prior  to 
Radng 


Three  Months 
After  Rating 


Six  Months  After 
Raring 


Date:  5/30/91 


8/30/91 


1/30/92 


4/30/92 


BCWS: 


BCWP: 


BCWP; 


BCWP: 


BCV\^: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  VolatiJity  index: 


LRE  Volarilfty  Index: 


Percent  Complete:  |  0,9393 


BCWSAcriwty:  f  -0.2897 


BCWPActMty:  |  -0.2897 


ACWP  Activity: 


Schedule  Performance  Index: 


Cost  Performance  Index: 


1.16667 


Investigator  Comments: 

i**INVALiD  DATA  POINT*"  Accumulated  costs  (ACWP.  BCWP)  moved  from  this  project  during  the  period  of  interest.  Invalidates 
i calculation  of  performance  indices. 


Data  identification 


Ra«ngTag:  [s'  'wBS#:  |2  1 

Specilication  design  and  integration  oversight  tasks.  Code  and  unit  test  of  database  architecture 


prgTag:  IT 


Rating  Information 


Rating  Date: 


Moderating  Variables 


Acquisition  Phase:  [Upgrade 


Contract  Type:  1  Other 


Program  Comments:  Icontract  converted  fron^  FPI  to  FPl/CPFF  during  this  period 


;S/W  Lifecycte:  .  (Multiple 


Language  %:  (100,00%  Appifca^on:  (Database 


Project  Budget: 


5015000  Budget  Volatility:  [Low 


45300 


%  New/Modified  Code; 


Requirements  Volatility:  (Low 
Cost  Accounting  Anomalies:  T 


Qualify  Stds  On  Contract:  3^  Qualify  Params  Tracked  : 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


e/30/91 


1/30/92 


4/30/92 


BCWP: 


BCWP: 


ACW>: 


ACWP: 


ACWP: 


Budget: 


Derived  Moderators 


Volatility  Index: 


Percent  Complete:  .  I  0,8618 


BGWS  Activity:  |  0.24093 


BCWP  Activity:  j  0.23739 


ACWP  Activity:  ]  0.25817 


Schedule  Performance  Index; 


Data  Identification 


OrgTag:  [T,  ftefingTag:  js'  WBS#:  |3__  ^ 

;  WBSDescription:  Isubsystem  test,  test  planning  and  integration 


Rating  information 

Rating  Date:  |  10/15/91  Rating: 

RateComment: 


Rating  Type: 


Rating  Relevance:  [High 


Moderating  Variables 


Acquisition  Phase: 


I  Upgrade 


Program  Comments: 


[contract  converted  from  FPI  to  FPi/CPFF  during  this  period 


SW  Lifecycle:  iTest 


Language  %:  i  0.00%  Application; 


15734000  Budget  Volatility: 


%  New/Modified  Code: 


Requirements  VolatSlity:  jlow 


Quality  Stds  On  Contract:  ^  Quality  Params  Tracked :  jj/ 


Cost  Accounting  Anomali^: 


Program  Manager  Comments: 


Six  Months  Prior  to 
Fating 


Three  Months  Prior  to 
Ra^ng 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


8/30/91 


1/30/92 


4/30/92 


BCWS: 


BCWS: 


BCWP: 


8CWP: 


12359 


ACWP: 


AONP: 


ACWP: 


ACV\^: 


Budget 


15219 


15734 


15740 


16050 


Derived  Moderators 


Budget  Volatility  Ind^:  J  0.04837  LRE  Voiatility  Index; 
BCWS  Activity:  I  0.30261  BCWP  Activity:  |  0.29784 


Percent  Complete:  1  07855 


ACWPActfvity:  I  0.30497 


Dependent  Variables 


Schedule  Perfonnance  index 


Cost  Performance  Index:  |  0.98186 


Data  Identification 

OfgTag:  jT”  ,  RatingTag:  |b^  WBS#; 
.  WBSDescripUon: :  loesign,  code  and  unit  test  of 


Rating  Information 

Rating  Date:  |  10/15/91  Rating: 

RateComment: 


Rating  Type:  ISCE 


Rating  Relevance:  jHigh 


Moderating  Variables 


Acquisition  Phase:  lUpgrade 


Contract  Type:  j Other 


Program  Comments:  Icontract  converted  from  FPl  to  FPi/CPFF  during  this  period 


Language  %:  j  100.00%  Application:  [Database 


17584000  Budget  Volatility; 


%  New/Modilied  Code; 


874300 


Requirements  Volatili^:  |Low 
Cost  Accounting  Anomalies:  f 


Quality  Stds  On  Contract:  Quality  Params  Tracked :  iyb 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  5/30/91 


1/30/92 


4/30/92 


14702 


15106 


12646 


BCWP: 


15757 


ACVS^: 


ACWP: 


14055 


16415 


Budget 


19066 


19889 


Derived  Moderators 


Budget  Volatility  index:  |  0.06933  LRE  Volatiit^  index:  |  0.1254  Percent  Co 

BCWS  Activity:  |  0.^264  ;  BCWP  Activi^:  I  0.27207  ACWP  ActivUy:  |  0.28956 


Dependent  Variables 


Schedule  Performance  Index; 


Cost  Performance  Index: 


Data  Identification 

OrgTag:  jT^  RatingTag:  p'  WBS#:  jT" 


WBSDescliption: 


Design,  code  and  unit  test  of  CSGls 


Rating  Information 

Rating  Date: 


10/15/91  Rating:  j 
RateCommenfc 


Rating  Type:  jjsCE 


Rating  Relevance;  jHigh 


Moderating  Variables 

Acquisition  Phase:  jupgrade 


Contract  Type:  |otner 


Program  Comnientsr  .  icontract  converted  from  FPl  to  FPl/CPFF  during  this  period 


S/W Lifecycle:;  Language:  jAda  ~  ~  ■  Language %:  |l 00,00%  Application:  |Database  • 

,  Project  Budget:  |  3953000  Budget  Volatility:  |low  Size:  j  78700  %  New/Modified  Code:  j  68.00% 

Requirements  Volatility:  |low  Rebaselining  :  |Nc  _ 


Quality  Stds  On  Contract:  y!!  Quality  Params  Tracked  : 


Cost  Accounting  Anomalies:  [ 

■■  1 

.  Program  Manager  Comments: 


Cost  Data 

Six  Months  Prior  to 

Rating 

Three  Months  Prior  to 

Rahng 

Three  Months 

After  Rating 

Six  Months  After  , 

Rating 

ill 

.i 

Date: 

1  5/30/91 

Date:  | 

8/30/91 

Date: 

L 

Date: 

[  4/30/92  ^ 

BCWS: 

1  3041 

BCWS; 

3354 

BCWS: 

[  3722 

BCWS: 

1  3848 

BCWP: 

1  3017 

BCWP: 

.  .  ....  .. 

32^ 

BCWP: 

[  3595 

BCWP: 

j  3810 

'  "  ? 

ACWP: 

1  2973 

ACWP: 

3223' 

ACWP: 

1  3664 

ACWP: 

j  3946:^ 

Budget 

1  3964 

Budget 

40iT 

Budget 

[  4023 

Budget : 

1  3953’’:; 

'  LRE: 

1  3981 

LRE:  1 

1  . 

4055 

LRE: 

[  4097'; 

LRE: 

1  4118;;"' 

Derived  Moderators 

-■  Budget  Volabiity  Ind®c:  |  -0.0028  LRE  Volatlli^  Index:  j  0.034  |  0.S638 

BCVl^AcSvify:  |  0.20972  BCWP  Activily;  |  0.20814'  ACWPActjv%:  |  0.24658  -• 


Dependent  Variables 

Schedule  Performance  Index:  |  _o.9S2652 
Investigator  Comments: 


Cost  Performance  Index:  j  o.sisoi; 


Data  Identification  >  ’  ; 

OrgTag:  RafingTag:  JIT  WBS#:  j[6_  . 

vySSbescription:  Isoftware  maintenance.  Design,  code  and  unit  tesl 


Rating  Information 

Rating  Date:  |  10/15/91 


Rating  Type:  jSCE 


Rating  Relevance:  [High 


Moderating  Variables 


Acquisition  Phase:  Jupgrade  Contract  Type:  j< 

Program  Comrnents:  ..  Icontract  converted  from  FPI  to  FPI/CPFF  during  this  period 


SW  lifecycle:  iMultipie-tate 


Application:  lOatabase 


1871 000  ;  Budget  Volatility : : :  I  Low 


%  New/Modified  Coc^; 


Quality  Stds  On  Contract:  ly!  Quality  Params  Tracked :  3^ 


Requirements  Volatility:  [Low 


Rebaseiinlrtg :  I  Ho 


Cost  Accounting  Anomalies; 


Three  Months 
After  Rating 


Six  Montibs  Prior  to 
Rating  \ 


Three  Months  Prior  to 
Raring 


Six  Months  After 
Faring 


Date:  I  5/30/91 


8/30/91 


1/30/92 


4/30/92 


BCWP; 


BCW>: 


ACWP: 


ACWP: 


ACWP: 


ACWP; 


Budget 


Derived  Moderators 


Budget  Volatili^  Index:  j  0.74209  LRE  VofatilHy  Index:  I  0.6253 


Percent  Complete:  |  0.0764 


BCWS  Acrivity: 


BCWP  Activity; 


ACWPAcrivity: 


0.704433 


Schedule  Performance  Index: 


Cost  Performance  Index:  |  1  02878 


Investigator  Comments: 


Data  identification 

OrgTag:  jF  feitingTag:  WBS#:  jT 

WBSDescription: 


Software-Related  management  activities:  Baselining,  Software  development  plartning,  etc 


Rating  Infonmation 

Rating  Date:  |  3/15/93. 


RatiingType:  ISCE 


Rating  Relevance:  iHigh 


Moderating  Variables 


Acquisition  Phase:  lUpgrade 


Contract  Type: 


Program  Comments:  Iccntract  FPI/CPFF 


S/W  lifecycle;  j  Multiple-Early 


Language%:  j  0.00%  Application: 


2553000  ■  Budget  Volatility :  ■  Low 


%  New/Moditied  Code 


Requirements  Volatility:  iMed 


Quality  Stds  On  Contract: 


Quality  Params  Tracked 


Cost  Accounting  Anomalies:  lEffort  is  winding  down 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating  .  : 


Date:  |  9/30/92 


Date:  I  12/30/92 


5/30/9: 


BCWS: 


BCWP; 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACW>: 


2558  ;  Budget 


Budget 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index:  I  0.0236 


0.04348 


BCWP  Activity:  I  0.04348 


ACWP  Activity:  I  0.04909 


Schedule  Performance  Index: 


Cost  Performance  Index: 


0.87402 


[Selected  for  model  validation, 


Data  identification 


RatingTag:  fC  WBS#:  E 


WBSPescnptipn:  r:-:  JSpecific^t^  design  and  integration  oversight  tasks.  Code  and  unit  test  of  database  architecture 


Rating  Information 

Rating  Date:  3/15/93  •  Rating: 

,  RateComment 


Rating  Type:  fSCE 


Moderating  Variables 


Acqufsib'on  Phase:  jupgrade 


Other 


I  contract  FPl/CPFF 


SW  Lifecycle:  iMultipie 


Language  %:  1 100.00%  Appiicaton:  IDatabase 


5142000  Budget  Volatility, 


54900 


%  New/ModllRed  Code: 


15.00% 


Requirements  Volatility:  |Low 
Cost  Accounting  Anomalies:  f 


Rebaselining :  iNo 


Quality  Stds  On  Contract:  art  Quality  Params  tracked :  a? 


Six  Months  Prior  to 
Radng 


Three  Months  Prior  to 
.  Rahng 


ThreeMonths 
After  Rating 


Six  Mondis  After 
Rating" ' 


Date:  I  9/30/92 


Date:  '  '  I  12/30/92 


5/30/93 


8/30/93 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP; 


Derived  Moderators 


BudgetVolahrity  Index:  LRE  Volatility  Index:  |  0.0189  Percent  Co 

sews  Activity:  I  0.07132  ''  BCWP  Activity:  [  o"o7424  ACWPActhnty:  \  0.1 1794' 


[Selected  for  mode!  validation, 


Data  identification 


OrgTag:  .  ;  EtetingTag:  WBS#:  p 

WBSDescdp^on: 


iSubsystem  test  test  planning  and  integration 


Rating  Information 

'  Rating  Date:  f  3/15/S3 


Ratiinglype:  ISCE 


Rating  Relevance:  (High 


Moderating  Variables 


Acquisition  Phase:  (Upgrade 


Contract  Type: 


Other 


Program  Comments:  jcontract  FPl/CPFF 


SW  Lifecycle:  [Test 


Language  %:  I  0.00%  •  Application:  [Database 


15867000  :  BudgetVolatjlity:  [Low 


%  New/Modified  Code: 


Requirements  Voiadiity:  I  Low 


Rebaselining :  [r^ 


Quality  Stds  On  Contract  ly!!  Quality  Params  Tracked  : 


Cost  Accounting  Anomalies: 


Program  Manager  Comments: 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  12/30/92 


8/30/93 


14761 


15730 


BCWP: 


14204: 


BCWP: 


ACWP: 


13708 


ACWP: 


14388 


ACWP: 


15126 


ACWP; 


15958 


15857 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index; 


Percent  Complete:  [  0.9875 


BCWSActJvify:  |  0.09224 


BCWPActrvrfy:  j  0.09344 


ACWPActivify:  I  0.11304 


Schedule  Performance  Index: 


Cost  Performance  Index: 


0.83801 


Invesdgator  Comments: 


Selected  for  model  validation. 


Rating  Information 

Rating  Date;  |  3/15/93  Rating; 

RateComment: 


Rating  Type:  ISCE 


Moderating  Variables 


Acquisition  Phase:  I  Upgrade 


Contract  Type:  iOther 


.contract  FPI/CPFF 


SAV Lifecycle:''  ll^ultiple 


Language  %:  1100.00%  Application:  IDatabase 


Project  Budgi^: 


1 8238000  Budget  Volatility:  iLow 


%  New/Modified  Code: 


1086000 


Quality  Stc^  On  Contract:  ^  Quality  Params  Tracked : 


Requiiements  Volatility:  ILow 


Rebasellning :  I  No 


Cost  Accoui^ng  Anomailes; 


Six  Months  Prior  to 
Rating 


Thf^  Montiis  Prior  to 

Rating-;'./^. 


Three  Months 
After  Rating 


Six  Months  After^ 
Rating 


Date:  f  9/30/92 


Date:  i  12/30/92 


18220  BCWS: 


BCWP: 


18181 


BCWP: 


18216 


ACWP: 


ACWP: 


ACWP: 


18238 


Derived  Moderate!^ 


Budget  Volatility  Index: 


LRE  Volatility  Index: 


BCWS  Activity:  I  0.04048 


BCWP  Activity:  I  0.0544 


ACWP  Activity:  0.08946 


Dependent  Variables 


Schedule  Perfonnance  Index: 


Cost  Performance  Index;  I  0-51427 


Data  Identification 

OrgTag:  JT^  RatingTag:  WBS  #:  [s**” 

WBSDescnptfon:  joes^gn"  code  and  unit  test  of  CSCIs 


Rating  Information 

Rating  Date:  |  3/15/93  Rating: 

RateComment: 


Rating  Type:  JsCE 


Rating  Relevance:  jHigh 


Moderating  Variables 

Acquisition  Phase:  |upgrade  .  Contract  Type:-  Jother  ^  ^ 

, Program, Gornments:  |a)ntractF^^  ^  ^ 

V,  SiWLif^cfe:  |Muitiple  ^  _  Language:  ^  ^  Language  %:  1 1 00.00%  Application:^.  |Database  ■ 

Project  Budget:  |  " ^51000 "  ■  .  Budget  Volatility:  jtow  :.Size:  j  "  'ssooo'  %  New/Modified  Code:  |  68  00% 

Requirements  Volatility:  ;  Rebaselining :  |no  Quality  Stds  On  Contract:  Quality  Params  Tracked  :  ST 

v  Cost  Accounting  Anomalies:  ■■■  |  .  . 


Program  Manager  Comments: 


Cost  Data 


Six  Months  Fiior  to 

Three  Months  Pnor  to 

Three  Months 

Six  Months  After 

Rating 

Rating 

After  Rating 

Rating 

Date:  |  9/30/92 

Date:  j 

12/S0/92 

Date:  j  5/30/93 

Date: 

j  8/30/93 

BCWS:  1 

3940 

BCWS: 

j  3952 

BCWS:  1  3951 

BCWS: 

J  395L 

BCWP:  1 

3937 

BCWP: 

1  3952 

BCWP:  j  3951 

BCWP: 

:j  3951 ; 

ACWP:  [ 

4167 

ACWP: 

1  4238 

ACWP:  1  4385/ 

ACWP: 

J  4436 

,  Budget:  | 

3951 

Budget: 

j  3952 

Budget:  |  3951 

Budget : 

1  3951! 

>  r~ 

4217  ' 

LRE: 

1  4273 

LRE:  [  aW  ■ 

'  LRE: 

1  4436; 

Derived  Moderators 

Budget  Vplatillty  Index:  |  0^ -vv  LRE  Volatility  Index:  |  0.0519  Percent  Complete:  |  ”  1  ■ 

BCWS  Activity:  j  0^0278;\  BCWP  Activity:  poODSsT  ACWP  Activity:  |  0.06064 

Dependent  Variables  . 

Schedule  Performance  Index:  [  L272727  Cost  Performance  index:  j 

Investigator  Comments: 

Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level 

7  '  '  '  '  .  A~34  .  ’ 


Data  identification 


OrgTag: 


R^ngTag:  [o'  vi®S#:  jT” 


:;VVBSDescnptJon:?'’:  {Software  maintenance.  Design,  code  and  unit  test 


Moderating  Variables 


S/W  Lifecycle: .  [y  uitipfe-Late 


Language  %:  J  100.00%  Application:  {Database 


2521000  Budget  VolatiJity:  llow 


%  New^odlinied  Code; 


Requirements  Volatility:  {low 


Quality  Stds  On  Contract:  Quality  Farams  Tracked :  yt 


Cost  Accounttng  Anomalies; 


Six  Months  Pnor  to 
Rating 


Three  Mon^  Prior  to 
Radng 


Three  Months 
After  Rating 


Six  Moh^  After 
^  '  Rating 


Date:  1  12/30/92 


5/30/93 


8/30/93 


BCWP; 


BCWP: 


ACWP: 


ACWP: 


ACW: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Voladlity  Index; 


LRE  Volatility  Index: 


BCWSAch^^ 


BCWP  Activity:  (  0.51484 


ACWPActlvi^:  |  0.5S455 


Dependent  Variables 


Schedule  Performance  Index: 


1.015071 


Cost  Performance  index:  I  0.97696 


Data  identification 

OfgTag:  |!r  RatingTag:  pT  WBS#;  |T 
WBSDescrlption: 


iDevelop  requirements,  design,  code,  and  test  system  software 


Rating  information 

Rating  Date:  I  sTii/iF  Rating: 


RatogType:  ISPA(irvnr) 


RateComment:  iGovernment-sponsored  contractor  did  an  assessment  to  suggest  possible  process  improvements 


Moderating  Variables 


Acquisr^on  Phase:  |£y  D 


Program  Comments:  Follow-on  to  previous  similar  efforts 


Language  %:  j  100.00%  Appltca^on:  jCommand  Sc  Co 


SW  Lifecycle:  iRequirements 


Jovial 


74S8000  Budget  Volatility:  I  Low 


%  New/Modified  Code:  I  100.00% 


148000 


Quality  Params  T racked :  ^ 


Quality  Stds  On  Contract: 


Cost  Accounting  Anomalies:  [None 


Program  Manager  Comments:  Beat  target  sched.  Had  experience  with  previous  similar  project,  but  subcontracted  the  software 

development.  Fell  behind  early  in  project,  but  instituted  process  improvement  initiatives  and  got  well  Size  in 
DSI 


Six  Months  After 
Rating 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rahng 


Three  Months 
Alter  Rating 


3/30/89 


Date:  I  12/30/88 


Date:  I  4/30/88 


7/30/88 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget: 


Budget  VoMlity  Index: 
:  BCWSAcfivity:  J  “ 


LRE  Volatility  Index:  0.0005 


ACWP  Activity; 


BCWP  Activity: 


Dependent  Variables 

Schedule  Performance  Index; 


0.73037 


Cost  Performance  Index: 


Investigator  Comments: 


Values  for  minus  6  month  and  minus  3  month  Budget  and  LRE  are  from  Dec  88  CPR.  This  was  done  to  avoid  DIV  0  errors  for 
derived  moderators.  Program  initiated  when  organization  was  rated.  Data  representative  of  12  months  after  rating  only 


Data  Idenfificatiori  ' 

OrgTag:  p^^  '  RaSngTag:  pT  WBS#:  |2  ’  ly 

WBSDeTCription:  ■  .joeveiop  requirements,  design,  code,  and'test  system  software 


Rating  Infomiaflon 

Rating  Date:  |  1  "  Rating  Type:  |SPA_(!NT)  Rating  Reievance:  |Med  __ 

'  "j;  .  RateComment  j Government-sponsored  contractor  did  an  assessment  to  suggest  possible  process  improvements 


Moderating  Variables 


Acquisition  Phase:  |£MD 


Program  Comments: I  Follow-on  to  previous  similar  efforts 


S/W  Lifecycle;  iTest/Integration  Language:  I  Jovial 


Language  %:  1 100.00%  '  Application:  iSimuiation 


2557000  Budget  Volatflity:  jLow 


%  New/Modiiied  Code: 


42000 


Requirements  Volatility:  jLow 
Cost  Accounting  Anomalies:  f 


Rebaselinlng :  I  No 


Quality  Stds  On  Contract: 


Quality  Params  Tracked : 


Progmm  Manager  Comments:  Beat  target  sched-  Had  experience  with  previous  similar  project,  but  subcontracted  the  software 

'  '  '  '  ' development.  Fell  behind  early  in  project,  but  instituted  process  improvement  initiatives  and  got  well.  Size  In 

'  D$i 


six  Months  Piiorto 
Rating 


Three  Montihs  Prior  to 
Rating 


Three  Months 
After  Rating 


Date:  i  4/30/88 


7/30/88 


Date:  I  12/30/88 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  Volatility  Index: 
BCWS  Actlvlfy:  I 


LRE  Volatility  Index: 


BCWP  Activify: 


ACWP  Activity: 


Dependent  Variables 

Schedule  Performance  index: 

InvestigatorComments:  V  - 


0.302778 


Cost  Performanice  Index:  I  1.01869 


Values  for  minus  6  month  and  minus  3  month  Budget  and  LRE  are  irom  Dec  88  CPR.  This  was  done  to  avoid  DIV  0  errors  for 
derived  moderators.  Program  Initiated  when  organiza^on  was  rated.  Data  representative  of  12  months  after  rating  only 


Data  Identification 


OrgTag:  ^  RafingTag:  |r'  'W®S#:  |p  ~  ■ 

V\®SDescription:  loeveiop  requirements,  design,  code,  and  test  system  software 


Rating  information 

Rating  Date:  I  3/iSs8  Rating; 


Rating  Relevance:  \Ue6 


RateComment  .  iGovemment-sponsored  contractor  did  an  assessment  to  suggest  possible  process  improvements 


Moderating  Variables 


Acqutsifeon  Phase: 


iFollow-on  to  previous  similar  efforts 


S/W  Lifecycle:  IRequirements 


Language  %:  1 100.00%  Appllca^on:  JCommand  &  Co 


3283000  Budget  V<^atlllty:  ILow 


%  New/Modi^ed  Code; 


Requirements  Volatility:  ILow 


Quality  Params  Tracked :  y!! 


Quality  Stds  On  Contract 


Cost  Accounting  Anomalies:  jSubcontracting  plan  did  not  materialize-thus  more  effort  expended  than  budgeted 


Beat  target  sched.  Had  experience  with  previous  similar  project,  but  subcontracted  the  software 
development.  Fell  behind  early  in  oroject,  but  instituted  process  Improvement  initiatives  and  got  welL  Size  in 
DSI 


Six  Months  Pnorto 
Rating 


Three  Months  Pnor  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  4/30/88 


7/30/88 


3/30/89 


BCWP: 


BCWP: 


BCWP: 


BCWP; 


ACm>: 


ACWP: 


ACWP: 


Derived  Moderatois 


Budget  Voladlii^  Index: 


LRE  Volatility  Index:  I  -0.0003 


Percent  Complete:  I  0.1376 


BCWS  Activity: 


BCWP  Achvity: 


ACWP  Activi^: 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index: 


1.07876 


lnv€^gator  Comments: 


Values  for  minus  6  month  and  minus  3  month  Budget  and  LRE  are  from  Dec  68  CPR.  This  was  done  to  avoid  DIV  0  errors  for 
derived  moderators.  Program  initiated  when  organization  was  rated.  Data  representahve  of  12  months  after  rating  only 


Data  Identification  , 

OrgTag:  Jir,;,  RatingTag:  |b'  '.'"WBS#:  |l  ' 

,  WBSDescripfion:;  loevelop  requirements,  design,  code,  and  test  system  software 


Rating  Information  ; 

Rating  Date:  |  4/15/91  Rating: 

RateComment: 


Rating  Type:  ISCE 


Moderating  Variables 


Acquisition  Phase:  lEMD 


Program  Comments:  |Fo(fow-on  to  previous  similar  eiforts 


S/W  Ufecycte: ;  iTest/lnteg  ration  .Language:  I  Jovial 


Language  %:  1 1 00 , 00%  >  <  "•  Application: ■  >  iCommand  &  Co 


7998000  Budget  Voladlity ; 


%  New/Mpdified  Code:  I  100.00% 


148000 


Rebaselinfng:  iNo 


Quality  Stds  On  Contract 


Quality  Params  Tracked : 


Program  Manager  Comments:  iBeat  target  sdhed.  Size  In  DSl 


Six  Montis  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Montis  After 
Rating 


Date:  I  10/30/90 


1/30/91 


6/30/91 


9/30/91 


BCW^: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP; 


Derived  Moderators 


Budget  Votatifity  index:  |  0.00858- LRE  Volatility  Index:  |  0.W87 

BCWS  Activity:  I  0.18467  ,  BCWPActivl^:  j  0.16613  ACWP  Activity: 


Percent  Complete:  |  1.0003 


Dependent  Variables 


Schedule  Performance  Index; 


1.06747 


A-39 


Data  Identification 

OrgTag.^  j!!r  Ratingtag:  p'  WBS#:  (T 
WBSDescription; 


iDevelop  requirements,  design,  code,  and  test  system  software 


Rating  information 

Rating  Date:  |  4/16/91  Rating: 

RateComment: 


Rating  Type:  ISCE 


Rating  Relevance:  flVIed 


Moderating  Variables 


S/W  Lifecycle:  ■  jTest/Integration  ..  Language:  -[jovial 


Language  %:  1 100.00%  Application:  I  Simulation 


2654000  Budget  Volatility:  flow 


%  New/Modified  Code: 


42000 


Requirements  Volatility:  [Low 


Rebaselining :  INo 


Quality  Stds  On  Contract: 


Quality  Pararns  Tracked  :  yC 


Program  Manager  Comments:  [Beat  target  sched.  Size  in  DSt 


Six  Montiis  Prior  to 
Rating 


Ttiree  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Date:  j  10/30/90 


1/30/91 


9/30/91 


BCWP; 


ACWP: 


ACWP: 


ACWP; 


ACWP: 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  index: 


BCWP  Acti\nty : .  j  0.1 6497 


ACWP  Activity:  I  0,0988 


Dependent  Variables 


WBSD^criptionr  IDevelop  requirements,  design,  code,  and  test  system  software 


Rating  Information 

Rating  0^::|  4/30/91  •'  Rating; 

.  RateComment: 


Ra^ngType:  ISOE 


Moderating  Variables 


Acquisition  Phase:  lEMD 


Program  Ccmiments:  iFoilow-on  to  previous  similar  efforts 


S/W  Ufecycfe:' ,  iTest/Integration  Language:  [Fortran 


Language  %:  1100.00%  Application:  ICommand  &  Co 


3432000  Budget  Volatility:  I  Low 


%  New/Moditied  Code: 


141000 


Requirement  Volatility:  I  Low 


Rebaselfning :  jiMo 


Quality  Stds  On  Contract: 


Quality  Params  Tracked : 


Cost  Accounting  Anoma!!^; 


Progi^m  IWfenager  CommentSf;;:^^  jSeat  target  sched.  Size  in  DSl 


Six  Months  Prior  to 
Rating 


Three  Monte  Prior  to 


Three  Months 
After  Rating 


Date:!  10/30/90 


1/30/91 


6/30/91 


9/30/91 


BCWP: 


BCWP; 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index:  I  0.0029 


Percent  Complete:  \  I.OOO: 


BCWS  Activity:  I  0.12325 


BCV^  Activity:  I  0.16108 


ACWP  Activity:  I  0.07298 


Dependent  Variables 


Schedule  Performance  index; 


Cost  Performance  Index: 


2.16016 


Selected  for  mode!  validation, 


A-41 


Data  Identification 


OrgTag:  RatingTag:  -Jc 

;=WBSO€Scnption:  ::-  Develop  recuirements,  design,  code,  and  test  system  software 


Rating  Information 

Rating  Date:  |  11/15/91  ^  Rating; 

RateComment 


Rating  Relevance:  [High 


Moderating  Variables 


Acqulsl^on  Phase:  lEMD 


[Fonow“On  to  previous  similar  efforts 


Program  Comments: 


S/WLif^cle:  llntegratior! 


Jovial 


Language  %: :  1 100.00%  Appllca^on:  ICommand  &  Co 


7998000  Budget  Volatiitty:  iLow 


%  New/Modified  Code:  |  1 00.00% 


Project  Budget: 


148000 


Quality  Params  Tracked  : 


Quality  Stds  On  Contract: 


Requirements  VolatjlHy:'  iLow 


Cost  Accoundng  Anomalies:  Very  little  effort  over  the  period  of  Interest-Actuals  over  period  only  .3%  of  actuals  to  date-will  affect  CPI 


•Program  Manager.;, Comments; . .  [Beat  target  sched.  Size  in  DSl 


Six  Months  Prior  to 
Rating 


Three  Mc^ths  Prior  to 
Rating 


Three  Months 
After  Rahng 


Six  Months  After 
Rating 


S/3D/91 


Date:  5/30/91 


4/30/92 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


LRE  Volatility  Index:  I  0.0011 


Budget  Volatility  Index: 


Percent  Complete: 


BCWP  Activity:  f  0.02863 


BCWS  Activity:  1  0.01825 


ACWP  Activity:  0.00293 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index: 


Inves^gator  Comments: 


Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level. 


Identification 

OigTag:  p*  RabngTag;  ^  VVBS#:  p 
WBSDescription:--  joevetop  requirements,  design,  coi 


Rating  Information 

Rating  Date:  I  11/15/91 


Rating  Type:  ISCE 


Rating  Relevance:  iHigh 


Moderating  Variables 


Aoquisi^on  Phase:  lEMD 


Contract  Type:  IFPIF 


S/WLH^cle:  |lrttegration  Unguage:  jJovial  ~  ^  ‘language  %;  1 100.00%'  Application:  jsimulatlon 

;  Project  Budget:  |  _  2654000  ’  Budget  Volatility:  ___  Size:  |  T  %  New/Modified  Code:  |  52.00% 

,r  Require^nteVoIafflity:  |low  Rebaseiining ;  jNo .  Quality  Stds  On  Contract  G  Quality  Params Tracked:  S 

CostAc^untingAnomaHes:  Jno  effort  for  this  WBS  over  the  time  period  of  interest-may  affect  performance  indices 


Six  Mor^s  Prior  to 
Rating 


Three  Months  Pnor  to 


Three  Months 
After  Rating 


Six  Months  After 
Racing 


5/30/91 


8/30/91 


1/30/92 


Date:  I  4/30/92 


BCWP; 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Voladlity  Index: 
BCWS  Activity:  I  0.01846 


0  LREVolatilify  Index:  |  -0.0009  • 

BCWP  Activity:  |  0.01846  ,  ACWP  Activity; 


Dependent  Variables 

.  :  Schedule  Perfoiroance  Index: 


Cost  Performance  Index: 


Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level. 


Data  Identification 


OtgTa^:  p"  ;  :;  R^ngTag:  [o'  WBS#:  js 
WKDescrlpUon:  Develop  requirements,  design,  cc< 


Rating  Information 

Rating  Date:  |  11/15/91  Rating: 

RateComment 


Moderating  Variables 


Acquisition  Phase:  \EMD 


Program  Comments:  jFoflow-on  to  previous  similar  efforts 


S/W  lifecycle:  I  Integration 


100.00%  Appiica^on. 


Command  &  Co 


3432000  Budget  Volatility:  |  Low 


%  New/Modified  Code: 


Requirements  Volatility:  ILow 


Rebaselining :  I  No 


Quality  Stds  On  Contract: 


Quality  Params  T racked :  ^ 


Cost  Accounting  Anomalies:  iLittle  effort  for  this  WBS  over  the  time  period  of  interest-may  affect  performance  indices 


Program  Wanager  Comments:  jSeat  target  sched.  Size  in  DSi 


Three  iVlonths  Prior  to 
Ra^ng 


Six  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  I  5/30/91 


8/30/91 


1/30/92 


4/30/92 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Budget 


Budget 


Derived  Moderators 


LRE  Volatility  index:  -0.002 


Budget  Voiatirity  Index: 


BCWS  Activity:  I  0.01923 


BCWP  Activity:  I  0.0201 


ACWP  Activity:  I  0.00371 


Schedule  Performance  index 


Cost  Performance  index:  I  5.30789 


Investigator  Comments: 


;Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level. 


/■V' ' 


Data  Identification 


RafingTag:  lA  WBS#:  12 


WBSDescnption:  JSubsystem  architecture,  database  administration,  and  software  configuration  management 


Rating  information 

Rating  Date;  |  12/15/89  '  Rating: 

RateComment: 


Rating  Relevance:  ...iHIgh 


Moderating  Variables 


Acquisition  Phase:  iSupport/Upgrade 


Program  Comments: 


Language  %:  1  0.00%  Application:  iDatabase 


Project  Budget: 


8451 000  Budget  Volatilil^:  ILow 


%  Kew/ModIfiedI  Code; 


Requirements  Volatility:  ILow 


Rebaselining :  I  No 


Quality  Stds  On  Contract: 


Quality  Params  Tracked :  y! 


Cost  Accounting  Anomalies:  (No  +/-  three  month  data 


Six  Months  Poor  to 
Ra^ng 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Radng 


Date:  I  6/30/89 


5/30/90 


BCV\^: 


8CWP: 


BCWP: 


ACWP: 


AONP\ 


ACWP: 


ACWP: 


Budget 


Budget 


Budget  Volatility  Index:  |  0.1 3057 


LREVoladlity  Index: 


BCWS  Activity:  I  0.13939 


BCWP  Activity:  I  0.1^3 


ACWP  Activity:  I  0.14817 


Dependent  Variables 


Schedufd'Perfonnance  Index: 


Cost  Perfomiance  Index:  f  0-86803 


Investigator  Comments: 


iNo  data  for  plus/minus  three  month. 


Data  identification 

OrgTag:  jir  RafingTag:  jl'  WBS#:  js 
,  WBSDescripfibn:  joverali  mangement  of  software  development  effort 


Rating  Information 

Rating  Date:  j  1^16/89  Rating: 

RateComment: 


Rating  Type:  |SPA(INT) 


Ra^ng  Relevance:  jHigh 


Moderating  Variables 


Acquisition  Phase:  |Support/Upgrade 
Program  Comments:  T  " 


Contract  Type: 


SIW  Lifecycle::'-^  ' jMultiple  ^  Language: :  |n/A 

Project  Budget:  |  ^  3205000"  -  Budget  Volafelity:  |low 

Requirements  Volatility:  Ilow  Rebaselining :  Wc 


0.00%  Application:  jOalabase 


%  NeAv/Modified  Code; 


Ouafii^  Stds  On  C<mtract: 


Quality  Params  Tracked : 


Cost  Accounting  Anomalies:  iNo  +A  three  month  data 


Three  Months  Prior  to 
.Rating 


Three  Months 
After  Rating 


Six  Months  After  ^ 
Rating 


Date:  I  6/30/89 


5/30/90 


BCWP: 


BCWP: 


^  ACWP: 


ACWP: 


ACWP: 


ACVS^: 


2237  ,  Budget 
2334  '  LRE: 


Budget 


Derived  Moderators 


Budget  Volatility  Index;  j  ,J?;^272  LRE  Volatility  Index:  |  0.4357  Percent  Complete;  |  0.8_811 

BCWSActiv%:  I  0.28293  ,  BCWP  Activity:  I  0.28293  ACWP  Activity:  |  0.24056 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index:  I  1.21799 


:No  data  for  plus/minus  three  month. 


Data  identification 


RaBngTag;  Ja'  -;V\®S#:  -J.' 

Requirements,  design,  code,  and  test  of  system  control  C$C1 


Rating  Information 

Rating  Date: .  I  1^15/89  .  Rating; 

,  "  RateComment: 


Ra^ng  Type:  >  ISPA  (IMT) 


Moderating  Variables 


Acqulsi^on '  Phase:; ; ;  ISupport/Upgrade 


Program  Comments; 


SW  Lifecycle:  IMultiple 


Language  %:  1 100.00%  ' '  Application:  IDatabase 


Project  Budget: 


2440000  Budget  Volatility:  flow 


%  New/Modsifed  Code; 


22400 


Requimments  Volatilify:  ILow 


Rebaselfning:  jNc 


Quality  Stds  On  Contract: 


Quality  Pamms  Tracked :  y; 


Cost  Accounting  Anomalies:  iNo  -fA  three  month  data 


Six  Montte  Prior  to 
Radng 


Thiw  Months  Prior  to 
Radng 


Three  Months 
After  Rating 


Date:  I  6/30/89 


5/30/89 


BCWS: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP; 


Derived  Moderators 


Budget  Voladirty  Index:  I  0.01035  LR£  Voladlity  Index: .  I  0.0104 


Percent  Complete:  I  0.9902 


BCWS  Activity:  I  0.11557 


BCWPAcdvfty;  I  0.10596 


ACWPActfvily:  I  0.07541 


Dependent  Variables 


Schedule  Performance  Index: 


0.907801 


Cost  Performance  Index:  I  145455 


’  A-47 ' 


Data  identification 

OrgTag:  jlT  RafingTag:  WBS#:  |5 


WBSDescriptiori: 


Requirements,  design,  code,  and  test  of  systems  interface  CSC! 


Rating  Information 

Rafing  Date;  |  12/15/89  ,  Rating: 

RateComment: 


Moderating 

Acquisition  Phase:  |$upport/Upgrade 


Rating  Type:  |SPA  (INp  Rating  Relevance:  |High 


Contract  Type:  IFPIF 


Program  Comments: 


S/W  Lifecycle:  Language:  |Fortran  "  ^  Language%:  |l0a00%  Application:  joatabase 

Project  Budget:  |  423S000  Budget  Volatility:  jTow  Size:  j  "  ^  ^  43200  %  New/Modified  Code:  j[ 

Requirements  Volatilii^:  |low  Rebaselining :  Quality  Stds  On  Contract:  CJ  Quality  Params  Tracked : 

Cost  Accounting  Anomalies:-  j  No three  month  data  "  *  ~  "  "  iiiii&i 

Program  Manager  Comments: 


Cost  Data 


Six  Months  Prior  to 
Rating 

Three  Montte  Prior  to 
Rating 

Three  Months 

After  Rating 

Six  Months  After 
Rating 

Date:  | 

6/30/89 

Date:  J 

- - 

Date:  j 

— ;/  , 

Date:  | 

5/30/90  ^ 

BCWS:  1" 

2286 

BCWS:  [ 

0: 

BCWS:  1 

0 

BCWS:  P 

3266: 

BCWP:  1" 

2279  :  A 

BCWP:  [ 

0 

BCWP:  j 

W 

BCWP:  1“ 

3167 

ACWP:  I" 

2190 

ACWP:  [ 

0 

ACWP:  j 

. 0 

ACWP:  1“ 

2989: 

Budget:  jf 

2581 

Budget  | 

0 

Budget  1 

0 

Budget :  j" 

4238: 

2515 

LRE:  1 

0 

LRE:  I 

b 

LRE:  [■ 

4169: 

Derived  Moderators 

Budget  Voladtity  Index:  J 

0.642: 

LRE  Vofablity  Index:  J  0.6577 

Percent  Complete: 

1  O'W? 

.BCWSAcfiwtif:  I  0.30049  -  BCVS/P  Activity;  |  0.28039  _  ACWP  Activity:  |  0.26731 


Dependent  Variables 

Schedule  Performance  Index: 

Inve^gator  Cc«fnments: 


n: 


904277 


Cost  Performance  Index:  )  i-iii39 


No  data  for  plus/minus  three  month. 


Data  Identification 


WBSDescription:  iRequirements,  design,  code,  and  test  of  applications  CSC! 


Rating  Information 

Rating  Date:  |  12/15/89 


Rating  Type:  |SPA{1NT) 


Rating  Relevance:  jHigh 


RateComment 


Moderating  Variables 


^  Acquisl^on  Phase:  jSupport/Upgrade 
Program  Comments:  I 


I  Fortran 


Language  %:  1 100.00%  Application:  IDatabase 


Project  Budget:  j  2683000  BudgetVolahliiy:  |Low 
Requirements  Voladlity:  |low  ^  ,  Rebaselming : 

Cost  Accounting  Anomalies:-  Jno  h^/-  three  month  data 


73200  %  New^odilRed  Code; 


Quality  Stds  On  Contract 


Quality  Params  Tracked : 


Program  Manager  Comments: 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rahng 


Six  Monft^  After 
Rating 


5/30/90 


2418 


BCW>: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget Voladlity  index:  |^Ja066^  LR£  Volatility  Index: 

BCWSActMty:  |  0.09653'  BCWP  Activity: 

Dependent  Variables  t  f 

Schedule  Peifonmance  index:  ,  j  o.9i5058'  : .  ’  - 


).056  Percent  Complete:  |  0.9896 

ACWPActi>rf^i  J  0.05104 


Cost  Performance  Index:  i  1 -75555 


No  data  for  plus/minus  three  month. 


Data  Identification 


WBSDescrtption:  [Requirements,  design,  code,  and  test  of  database  maintenance  C$C1 


Rating  Information 

’  Rating  Date;  |  12/15/89 


Rating  Type:  jSPA  (INT) 


Rating  Relevance:  [High 


Moderating  Variables 


IFortran 


Language  %:  [100.00%  Application:  (Database 


Project  Budget: 


2667DOO  Budget  Volatility 


%  Nevw/Modffied  Code; 


Requirements  Volatility:  [Low 


FtebaseJining :  [No 


Quality  Stds  On  Contract: 


Quality  Params  Tracked  : 


Cost  Accounting  Anomalies:  [No  '‘■A  three  month  data 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Mondts 
After  Rating 


Six  Months  After 
Rating 


5/30/90 


BCWS: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Budget 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index:  |  -0.039 


BCWSAcdvItyrf  0  06189 


BCWP  Activity:  I  0.06113 


ACWP  Actlvi^:  f  0.02756 


Schedule  Performance  Index: 


Cost  Performance  index;  I  2.05063 


Invesdgator  Comments: 


No  data  for  plus/minus  three  month. 


Data  Identification 


O^Tag:  RafingT^:  jj^  WBSft  |8 

WBSDescriplion:  iRequirements,  design,  code,  and  test  of  database  support  CSCI 


Rating  information 

Rating  Date:  J  12/15/89*  Rating: 

RateComment: 


Rating  Relevance:  jHigh 


Moderating  Variables 


Acquisition  Phase: 


Contract  Type: 


Language  %:  1100.00%  Application:  -(Database 


Project  Budget:  |  1181 000  :  Budget  Volatility :  (Low 

Requirement  Voladlity:  |low  Rebaselining  :  |no 

Cost  Accoundng  Anomali^:  :  Ino  +/-  three  month  data 


14200 


%  New/Modifted  Code: 


Quality  Stds  On  Contract: 


Quality  Params  Tracked :  ’SL 


Six  Months  Pnor  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


5/30/90 


BCWP: 


BCWP: 


BCWP: 


ACW; 


ACWP: 


ACWP: 


ACWP: 


Budget  Volatility  Index:  J  0^01635  LRE  Volatility  Index:  |  0.0119  Percent  Complete:  [~  a9949 

BCWSActiwty:  f  0,01106  BCWP  Activity:  j  0  01277  ACWP  Activity:  f  0.00632  .  V: 


Dependent  Variables 


Schedule  Performance  index; 


Cost  Performance  Index: 


Data  point  excluded  from  Complete  Data  Set  due  to  lov/  activity  level.  No  data  for  plus/minus  three  month, 


A-5} 


Data  Identification 

OrgTag:  :  F  ‘-:  ..  RatingTag:  |A 


WBS#:  |9 


WBSDescription:  ISoflware  mtegration  activities. 


Rating  Information 

Rating  Date:  | _ 1^15/89  .  Rating:  j" 

RateComment: 


Moderating  Variables  ; 

Acquisition  Phase:  |^Support/Upgrade 
Program  Comments: 


Rating  Type:  |sPA  (!fsnr)  Rating  Relevance:  jHigh 


Contract  Type:  IFPIF 


S/W  Lifecycle:  jTest/lntegration  Language:  |Fo^ran 
Project  Budget:  |  5821000  Budget  Volatility:  jT 


Low 


Requirements  Voladlity:  iLow 


Rebaselintng :  iNo 


Language  %:  1 100^ 00%'  .  ■  Application:  JoatabWe 
Size:  ’I 

Quality  Stds  On  Contmct: 


0  %  New/Modified  Code:  |  0.00%  ; 


Quality  Params  Tracked :  y; 


Program  Manager  Cc»fiments: 


jNo  three  month  data 

' ''  '  . . ■; . . 

•  s. 

Cost  Data 

Six  Months  Prior  to 
Radng 

Date:  [  6/30/89  ' ' 

BCWS;  I” 

BCWP:  j” 

ACWP; 


Three  Months  Prior  to 
:  Rating 


Three  Mondis 
After  Rating 


Six  Months  After 
Rating 


3009 


3002 


5287 


Date:  J” 
BCWS:  I" 
BCWP:  I” 
ACWP: 


Date:  f 
BCWS:  |[ 
BCWP: 


Date:  [ 
BCWS:  r~: 


4949 


Budget  j* 
LRE:  [ 


5928 


7906 


Budget  ^ 

t 


ACWP:  j 
Budget 


LRE: 


=  r 


BCWP:  J“ 
ACWP:  I" 
Budget : 
UFE:  [" 


4784 


7574 


5821 


8375 


Derived  Moderators 

Budget  Volatility  index:  |  -0.018  LRE  Volatility  Index: 

BCWSActiv%:  0.392  ... BCWP  Acthnfy:  j  0.37249 

Dependent  Variables 

Schedule  Performance  index: 

Inveshgator  Comments: 


0.918557 


^0593  Percent  Complete:  |  0.8219j: 

.  ACWPActivi^:  |  0.30195 


Cost  Performance  Index:  f 


No  data  for  pius/minus  three  month. 


A-52 


Data  Identification . 


OrgTag:  jfC-  f^tingTag:  \A©S#:  J2”  ^ 

-  WBSDescriplion;  ,,,  jSubsystem  architecture,  database  administration,  and  software  configuration  management. 


Moderating  Variables 


Acquisition  Phase: 
Program  Comments: 


SW  Lifecycle:'  iMultiple 


Laji^aQ&%:  I  0.00%  Application:  I  Database 


B586000  -r  Budget  Volatilify:  iLow 


%  New/Mods^ed  Code: 


Requirements  Volatili^:  |Law  Rebaseltnii 

■Cost: Accounting  Anomalies:  \ho  +A  three  month  data 


Quality  Stds  On  Contract: 


Quality  Params  Tracked :  :;/! 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  toj 
Rating  . 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


6/30/90 


Date:  I  11/30/90 


2/28/91 


BCWP: 


BCWP: 


ACWP: 


ACWP; 


ACWP: 


ACWP: 


0  Budget 


Derived  Moderators 


Budget  Volatility  Index:  [  0.01597 


Percent  Complete:  I  0.9888 


BCWS  Activity:  I  0.09738 


BCWP  Activity:  I  0.09929 


ACWP  Activity:  |  0.10264 


Dependent  Variables 


Schedule  Perfortnance  Index: 


Cost  Perfomia  nee  Index:  I  0.912M 


Investigator  Comments: 


No  data  for  plus/minus  three  month. 


Data  Identification 


OrgTag:  IK 


:.WBSDescriptlonr  ^- jOveran  mangement  of  software  development  effort 


Rating  Information 

Rating  Date;  |  _  9/15/90  I^ng: 

RateComment: 


Moderating  Variables 


Acquisition  Phase: 


Con^ct  Type: 


Program  Comments: 


S/W  Lifecycle:  feuitiple 


0.00%  Application:  (Database 


3239000  Budget  Volatility:  ILovv 


%  New/Modjfied  Code: 


Requirements  Volatility:  (Low 


Quality  Params  Tracked : 


Quality  Stds  On  Contract: 


Cost-Accounting  Anomalies:  |No  +/-  three  month  data 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


6/30/90 


Date:;]  11/30/90: 


2/2S/91 


BCWP: 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACVW>; 


ACWP: 


ACWP: 


Budget 


Budget: 


>205  Budget 


Budget  Volatility  index:  I  0.01 061 


LRE  Volatili^  Index: 


Percent  Complete:  j  0.99 1 4 


BCWS  Activity:  f  0.16568 


BCWP  Activity:  j  0.16568 


ACWP  Activity:  0.16271 


Dependent  Variables 


Schedule  Perfoitnance  Index: 


Cost  Performance  index: 


1.04931 


Investigator  Comments: 


No  data  for  plus/minus  three  month. 


Data  Identification  ,  :  • 

OrgTag:  jiT’  RaBngTag;  Ib*  WBS  #:  |4  - 

.  WBSDescription:  (Recjuirements,  design,  code,  and  test  of  system  control  CSCI 


Rating  inforrtiation 

Rating  Date:  |  9/15/90 


Rating  Relevance:  pigh 


Moderating  Variables 


Acquisltioh  Pha^: 
Program  Comments: 


S/W  Lifecycle:  llVIultipfe 


Language%:  h  00,00%  Application:  j Database 


Project  Budget: 


2440000  Budget  Voiatility:  ILow 


%  New/Modliied  Code: 


22400 


Requirements  Volatility:  |Law  Rebaselining  :  /  |No 

Cost  Accoundng  Anomalies: :  Ino  effort.  No  +/- three  month  data 


Quali^  Stds  On  Contract: 


Quality  Params  Tracked :  3!^ 


Six  Months  Prior  to 
Radng 


Three  Months  Pnorto 
Radng 


Three  Months 
After  Rating 


Six  Months  After 
Radng  ' 


Date:  |  3/30/90 


6/30/90 


BCW5: 


BCWS: 


BCWP: 


BCV\^: 


BCWP; 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


Derived  Moderators 


Budget  Voladlity  Index: 


LRE  Volatility  Index: 


Percent  Complete: 


BCWS  Act^ty: 


BCWPAcbVity:  I  0.00984 


ACWP  Activity: 


Dependent  Variables 


Schedule  Performance  Index 


Cost  Performance  Index: 


Data  IdentilicatioiTi 

OrgTag:  jlT  ,  Rj^ngTag:  WBS#:  js 

WBSDescription:  iRequirements,  design,  code,  and  test  of  systems  interface  CSCI 


Moderating  Variables 


Acquisiton  Phase: 


Program  Comments: 


S/W  Lifecycfe: 


Language: 


Language  %: 


100.00%  • 


Applicahon: 


4236000  ;r..  budget  Volatility :  iLow 


%  New/Modified  Code: 


43200 


Requirements  VolatiJity:  jLow  Rebaselining : 

Co^  Accounting  Anomalies:  Ino  +A  three  month  data 


Quality  Params  Tracked  : 


Six  Months  Prior  to 
Rating 


Three  Months  Piiorto 
Rating 


Three  Months 
After  Rating 


Six  Months  After  v 
Rating 


Date:  3/3C/90 


Date:  11/30/90 


2/28/91 


BCWP: 


ACWP: 


ACWP; 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Volatility  Index:  |  -0.0005  LRE  Volatility  Index: 

.  BCW®  Activity:  I  0.27219  BCWP  ActiviQr:  I  0.28033 


Percent  Complete:  ]  0.9903 


ACWP  Activity:  I  0.1 8739 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Performance  Index: 


No  data  for  pius/minus  three  month. 


A.r56 


Data  Identification 


OrgTag;  Ratingtag:  js'  WBS#:  |6  ;  /I  V 

'WBSpescripSon:  [Requirements,  design,  code,  and  test  of  appSications  CSCl 


Rating  Information 

Rating  Date:  |  9/15/90  Rating; 

■  RateComment 


Rating  Type:  ISCE 


Ra^ng  Relevance:  jHigh 


Moderating  Variables 


S/W  Lifecycle:  I  Multiple 


Language:  I  Fortran 


Project  Budget: 


2683000  Budget  Volatility:  [low 


R^ulrements  Volatility:  I  Low 


Rebasetining :  I  No 


Language  %:  j  100. 00%  Application:  |Datab3se 

Size:  |  73200  %  New/Modified  Code:  |  85.00%.  • 

Quality  Stds  On  Contract  Qualify  Params  Tracked :  -j 


■Cost  Accounting  Anomalies:  i  JNegligible  effort  during  this  period.  No  +A  three  month  data 


Program  Manager  Comments; 


Six  Mondis  Prior  to 
Rating 


Three  Months  Prior  to 
-  Rabng 


Three  Months 
After  Rating 


Six  Montis  After 
Ratng 


6/30/90 


Date:  I  11/30/90 


BCWS 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACVW>: 


Budget 


Derived  Moderators 


Budget  Volatilify  Index: 
BCWS  Acuity:  |  0.00534 


LRE  Volatilify  Index: 


Percent  Comj^^ete: 


BCWP  Activity:  I  0.0D525 


ACWPActivify:  I  0.00151 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index: 


Data  point  exduded  from  Complete  Data  Set  due  to  low  activity  level  No  data  for  plus/minus  three  month. 


Data  identification 


OrgTag:  pT  RatingTag:  p'  WBS#;  J? 
WBSDescnptron: 


Requirements,  design,  code,  and  test  of  database  maintenance  CSCl 


Rating  information 

Rating  Date:  |  9/15/90  Rating: 

RateComment 


Rating  Type: 


Rating  Relevance:  I  High 


Moderating  Variables 


Acquisition  Phase: 


Contract  Type: 


Program  Comments: 


Fortran 


1 00.00%  Application:  ^  ^  [Database 


2666000  BudgetVolatlllty: 


%  New/Modified  Code: 


Requirements  Volatility:  ILow 


Rebaselinfng 


Quality  Params  Tracked  : 


Quality  Stds  On  Contract: 


I  Negligible  etfort  during  this  period.  No  -f-/-  three  month  data 


Cost  Accounting  Anomalies: 


Six  Months  Prior  to 
Rating 


Three  Mondis  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


6/30/90 


11/30/90 


2/28/91 


BCWS; 


BCWS 


BCWP: 


BCWP: 


BCWP; 


ACWP: 


ACWP: 


ACWP: 


ACWP; 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index: 


BCWS  Activity: 


BCWP  Activrfy: 


ACWP  Activity:  |  0.00139 


Dependent  Variables 


Schedule  Performance  Index: 


Investigator  Comments: 


Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level.  No  data  for  plus/minus  three  month. 


Data  Identification 


OrgTag:  IK 


V^SDescrlption:  iRequirements,  design,  code,  and  test  of  database  support  CSC 


Rating  Information 

Rating  p^:  |  9/15/90  Rating: 

RafceComment 


Rating  Type:  ^CE 


Radhg  Relevance:  iHigh 


Moderating  Variables 


Acquisition  Phase: 


:SWlJfecycle:  ultiple 


Language  %:  1100,00%  Application:  iDatabase 


Project  Budget: 


1181 000  Budget  Volatility:  I  Low 


14200 


%  Mew/Modified  Code; 


Rebaseltning :  I  No 


Quality  $tds  On  Contract: 


Quality  Params  Tracked : 


Cost  Accminting  Anomalies:  I  Negligible  effort  during  this  period.  No  +/-  three  month  data 


Program  Manager  Comments: 


Six  Months  Prior  to 
Rating 


Three  Montiis  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


6.^0/90 


1/30/90 


BCWS: 


BCWP: 


BCWP: 


BCWP; 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


ACWP: 


0  Budget: 


Budget  Volatility  Index; 


LRE  Volatility  Index; 


Percent  Complete: 


BCWS  Activity:  I  0,00508 


BCWP  Activity:  I  0.00508 


ACWP  Activity:  I  0.00236 


Dependent  Variables 


Schedule  Performance  Index: 


Cost  Perfomiance  Index; 


Data  point  excluded  from  Complete  Data  Set  due  to  low  activity  level  No  data  for  plus/minus  three  month, 


Data  Identification 

OrgTag:  pT  Ra^ngTag:  |8^  WBS  #:  js" 


W^Descrlption: 


^Software  integration  activities. 


Rating  Information 

RatjngDate:  |  9/15/90  Rating:  |[ 

RateComment 


Rating  Type:  poT 


Rating  Relevance:  jhigh 


Moderating  Variables 


Acqufsrl^on  Phase:  |  "  Contract  Type:  j[ 


Program  Comments:  j .  ' 

if  ' 

..  , . ; . ,  - 

- .  ’  ’ 

'  A 

.  /  ■■  - r-™™-  . 

S/W  Lifecycle:  jXest/lntegration 

Language:  |Fortran 

Language%: 

1 100.00% 

Applicadon:  j  Database 

Project  Budget:  |  6874000 

Budget  Volatility:  |low 

Size:  j 

0  %  New/Modihed  Code:  |  85.00%  ' 

Requirements  Vola^ity; 

JLow 

Rebaselining : 

No 

Quality  Stds  On  Contract: _ 

Quality  Params  Tracked  :  ^ 

Cost  Accounting  Anomalies:  jNo  +/-  three  month  data 

■  .4 

Program  Manager  Comments: 

Cost  Data 

Six  Months  Prior  to 
Rating 

Three  Mondis  Pnor  to 
Rahng 

Three  Months 
After  Rating 

Six  Months  After 

Rating 

Date:  |  3/30/90 

Date:  1"  6/30/90 

Date: 

j  11/30/90 

Date: 

j  2/28/91 

BCWS:  1  4564 

BCWS:  [  0 

BCWS: 

1 . . 

BCV\^: 

1  6486 

BCWP:  1  4426 

BCWP:  j  0 

BCWP: 

1 _ 

T 

BCWP: 

j 

ACWP;  1  7084 

ACWP;  j  0 

ACWP: 

1,^ . 

“ 

ACWP: 

1  946'l  . 

Budget:  |  5821 

i  ■■  Budget:  |  O  ' 

Budget: 

1  ....  .. 

Budget : 

1 

LRE:  1  7384 

LRE:  1  0 

LRE: 

r.  iCf^E:  ;■ 

i  10014  ^ 

*  ''5 

Derived  Moderators 


Budget  Volatility  Index:  |  0.180^  LRE  Volatility  Index:  |  OSSsF  Percent  Complete:  |  0.9436 


BCWS  Activity:  j  0.29633  BCWP  Activity:  j  0.31761 

ACWP  Activity:  |  0.25124 

Dependent  Variables 

Schedule  Peiformance  index:  | 

Cost  Performance  Index:  [ 

0.86664: 

Investigator  Ccwnments: 

I  No  data  for  plus/minus  three  month. 


Data  Identification 


OrgTag:  pT  RafingTag:  WBS#:  |l  v 

WBSDescnpUon:  jcenerates  al!  sytem  design  requirements  (logic  &  algorithms)  and  software  to  support  technology  item  being  developed 


Rating  Information 

Rating  Date: 


/92  Rating:  |  2  Rating  Type:  JSPAJEXT)  Rating  Relevance:  jLow  ^ 

RateGornment:;  jconducted  in  accordance  with  an  SEi-Hcensed  vendor  agreement  between  *vendor  and  SEl 


Moderating  Variables 


Acquisition  Phase:  jConcept  Exploration  Contract  Type:  jCPI 

Program  ComTnents:  j85%  software,  1 5%  hardware”  Program  partially  terminated  after  technology  demonstrated 


Language  %:  [100.00%  '  Application:  [Avionics 


Project  Budget: 


2726000  iLow 


76636  %  New/IVIodIfied  Code:  {  100.00% 


Requirements  Volatility:  [Med  Rebasellning :  [No  Quality  Stds  On  Contract*  Z1  Quality  Params  Tracked 

Qost  Accounting  AnomaUes:  No  agreement  on  Estimate  to  Complete.  Contractor  may  have  tned  to  ''get  weir  on  options.  Contractor  took 
V-  -v oamcd  valuo  eafly 


Requirements  changes  due  to  interfaces  with  associate  contractor.  Overruns  covered  by  termination 
agreement.  Language  was  early  Ada  (non-validated  compiler).  Contractor  dted  too  much  documentation  as 
reason  for  overrun. 


Six  Months  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


:  Six  Months  After 
Rating 


Date:  I  12/30/91 


3/30/92 


8/30/92 


Date:  I  11/30/92 


8CWS: 


BCW>; 


BCWP: 


ACWP: 


ACWP: 


ACWP: 


Budget 


Derived  Moderators 


Budget  Volatility  Index:  |  0.00368  UF^  Volatility  Index:  [  0,  0012  Percent  Cc 

BCWS  Acfivity:  |  0.1799S  BCWPAcfivily:  {  0.14521  ACWPActivify:  |  0.12771 


Dependent  Variables 


Cost  Performance  index: 


Schedule  Performance  index; 


Data  Identification 


OrgTag:  JivT  RatrngTag:  pr '  WBS#:  |T 
WBSDescripfon: 


.Modify  existing  software  for  new  configuration 


Rating  Information 

Rating  Date:  I  10/15/92  Rating; 


Rating  Relevance:  jHigh 


RateGomment:  iPerformed  by  a  former  S51  employee:  "borderline* 


Moderating  Variables 


Acquisition  Phase:  lElVID 


Contract  Type: 


Program  Comments: 


SiW  Lifecycle:  I  Multiple-Early 


Language  %:  j  90.00%  Application:  iCommand  &  Co 


2230000  Budget  Volatility:  iLow 


550000 


%  New/Modllied  Code: 


Requirements  Volatility:  I  low 


Quality  Stds  On  Contract: 


Quality  Params  Tracked :  nl. 


Cost  Accounting- Anomaliesr:^;-  jlncreasing  baseline  reflected  througth  ECPs 


Program  Manager  Comments: 


Six  Months  Prior  to 
Rating 


Three  Months  Priorto 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


Date:  t  5/30/92 


8/30/92 


4/30/93 


BCWS: 


BCWP: 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


ACWP 


ACWP: 


2227  Budget: 


Derived  Moderators 


Budget  Volatillly  Index:  |  0.00135 


LRE  Volatility  Index: 


BCWP  Activity. 


ACWP  Activity: 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index; 


Data  Identification 


WBSDescriptlon:  Iwodity  existing  software  for  new  conliguration 


Moderating  Variables 


Acquisition  Phase: 
Program  Comments: 


S/WLif^cle:  IXest/Integration  :  I  Fortran 


Language  %:  I  90.00%  Appiicahon:  jCommand  8c  Co 


Project  Budget: 


2268000  Budget  Volatility:  I  Low 


%  New/Modified  Code: 


80.00% 


Requirements  Volatility:  Itow 


Rebaselining :  iNo 


Quality  Stds  On  Contract: 


Quality  Params  Tracked : 


Cost  Accounting  Anomalies::  llnaeasing  baseline  reflected  througth  ECPs 


Program  Manager  Comments: 


Six  Mon^  Prior  to 
Rating 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating 


6/30/93 


Date:  I  11/30/93 


BCWP: 


BCWP: 


ACWP: 


ACWP: 


2230  Budget: 


Derived  Moderators 


Budget  Volatility  Index: 


LRE  Volatility  Index:  I  0.0211 


Percent  Complete:  I  0.9951 


BCWS  Activity:  |  0  10714 


BCWP  Activity:  f  0.13735 


ACWPAc^tly:  I  0,19179 


Dependent  Variables 


Schedule  Performance  index: 


Cost  Performance  Index:  I  0.7711 4 


A-63 


Data  Identification 

OrgTag:  RatingTag:  ^  WBS  #:  p 

WBSDescrfption: 


Design,  code,  test,  integration  of  all  software  for  entire  system  consisting  of  3  major  components 


Rating  Information 

Rating  Date;  |  2/15/94  Rating: 

RateComment; 


Moderating  Variables 


Acquisi^on  Phase:  lEMD 


S/W  Lifecycle:  lOesign/Code 


Language  %:  1 100.00%  Appiicafen:  ISimuiation 


3153000  Budget  Volatilify:  iLow 


%  New/Modified  Code:  I  100.00% 


130000 


Requirements  Volatilify:  jMed 


Quality  Stds  Gn  Contract:  ^  Quality  Params  Tracked :  ^ 


Rebaselming :  iNo 


Cost  Accounting  Anomalies; 


Program  lyianager  Gc^ments:  [Company  does  not  have  domain  expertise.  ECPs  drivers  of  cost  growth 


Six  Mon^s  Prior  to 
Ra^g 


Three  Months  Prior  to 
Rating 


Three  Months 
After  Rating 


Six  Months  After 
Rating  , 


Date:  I  8/30/93 


Date:  I  11/30/93 


7/30/94 


BCWS: 


BCWP: 


BCWP: 


ACWP: 


AONP\ 


AONP 


ACWP: 


Budget: 


Derived  Moderators 


Budget  Volatility  index:  I  0.09138 


LRE  Volatility  Index:  j  0.5893 


Percent  Complete:  \  0.6952 


BCWS  Activity:  0.46959 


BCWPActivi^:  I  0.34717 


ACWP  Activity:  [  0.56818 


Schedule  Performance  Index: 


Cost  Perfomniance  index:  I  0.23626 


Investigator  Comments: 


Appendix  B:  Data  Supporting  Analysis  of  Complete  Data  Set 


This  appendix  contains  the  complete  set  of  plots  used  to  support  the  assumptions 
of  normality.  The  plots  were  constructed  by  the  statistical  software  package,  Statistixfor 
Windows. 


B-1 


1.  Box  Plots  of  CPI  and  SPI 


Box  and  Whisker  Plot 


RAUNG 


Figure  B-1  Box  Plot  of  SPI  for  Complete  Data  Set 


Box  and  Whisker  Plot 


Ordered  Data  Ordered  Data 


Wilk-Shapiro  /  Rankit  Plot  of  SPI 


-2-10  1  2 

Rankits 

Approximate  Wilk-Shapiro  0.9337 


B-5  Wilk-Shapiro  Plot  of  SPI  at  Rating  Level  2  for  Complete  Data  Set 


Wilk-Shapiro  /  Rankit  Plot  of  CPI 


-2-10  1  2 

Rankits 

Approximate  Wilk-Shapiro  0.9078 


Figure  B-6  Wilk-Shapiro  Plot  of  CPI  at  Rating  Level  2  for  Complete  Data  Set 


B-4 


Ordered  Data 


Wilk-Shapiro  /  Rankit  Plot  of  SPI 


-2 


-1 


2 


Rankits 

Approximate  WIk-Shapiro  0.9274 


Figure  B-7  Wilk-Shapiro  Plot  of  SPI  at  Rating  Level  3  for  Complete  Data  Set 


1.13 


1.06 


I  0.99 
Q 

*o 

O  0.92 


0.85 


0.78 


-2-1012 

Rankits 

Approximate  Wilk-Shapiro  0.9016 


Wilk-Shapiro  /  Rankit  Plot  of  CPI 


Figure  B-8  Wilk-Shapiro  Plot  of  CPI  at  Rating  Level  3  for  Complete  Data  Set 
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